Entrainment sonification techniques

ABSTRACT

A processing device in one embodiment is configured to generate a first sound cue of a first type, the first sound cue comprising a primary entrainment cue for an entrainment sonification system, to generate one or more additional sound cues of a second type, each of the one or more additional sound cues comprising an auxiliary entrainment cue for the entrainment sonification system, to provide the first sound cue and the one or more additional sound cues to one or more audio devices of the entrainment sonification system for generation of sound for audible presentation to a user, to receive from one or more sensors of the entrainment sonification system one or more feedback signals, and to adjust one or more characteristics of at least one of the first sound cue and the one or more additional sound cues based at least in part on the one or more received feedback signals.

RELATED APPLICATIONS

The present application is a continuation-in-part of U.S. patentapplication Ser. No. 16/634,928, filed Jan. 29, 2020 and entitled“Method and System for Musical Communication,” which is a 35 U.S.C. §371 national phase of PCT Application No. PCT/US2018/047375, entitled“Method and System for Musical Communication,” which was filed on Aug.21, 2018 claiming priority to U.S. Provisional Patent Application Ser.No. 62/548,001 filed Aug. 21, 2017, each of which is incorporated byreference herein in its entirety. The present application also claimspriority to U.S. Provisional Patent Application Ser. No. 62/802,521,filed Feb. 7, 2019 and entitled “Breathing Entrainment SonificationTechniques,” U.S. Provisional Patent Application Ser. No. 62/850,882,filed May 21, 2019 and entitled “Goal-Driven Auditory Display Techniquesfor Cardio Fitness, Aerobic Activity and other Contexts,” and U.S.Provisional Patent Application Ser. No. 62/915,935, filed Oct. 16, 2019and entitled “Applications for Musical Communication System,” each ofwhich is incorporated by reference herein in its entirety.

COPYRIGHT STATEMENT

A portion of the disclosure of this patent document contains materialthat is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

FIELD

The field relates generally to information processing systems, and moreparticularly to systems that implement techniques for musicalcommunication and/or sonification for entrainment and/or otherapplications.

BACKGROUND

A wide variety of different systems for musical communication and/orsonification are known to those skilled in the art. However, these andother conventional systems fail to provide suitable entrainmentsonification. For example, the conventional systems are highly limitedin their functionality and effectiveness. Accordingly, a need exists forimprovements in the field of sonification.

SUMMARY

Illustrative embodiments provide methods, apparatus, systems andcomputer program products for entrainment sonification. The disclosedtechniques are particularly well-suited for use in breathing entrainmentsonification but are more generally applicable to numerous othercontexts involving entrainment sonification.

In some embodiments, a breathing entrainment sonification system isimplemented in the form of a musical communication system moreparticularly configured for a breathing entrainment sonificationapplication. A wide variety of other entrainment sonificationembodiments are possible utilizing the techniques disclosed herein.

For example, in some embodiments, a breathing entrainment sonificationsystem is configured in accordance with a breathing entrainment model.The system configured in accordance with the breathing entrainment modelmore particularly comprises a closed-loop system featuring two differenttypes of sound cues, namely, a sound cue of a first type to direct theuser's breathing pattern (an “entrainment component”) and one or moresound cues of a second type to provide feedback to the user on theircurrent status during the exercise (one or more “auxiliary components”).

Other types of closed-loop and open-loop entrainment sonificationsystems, each utilizing at least one entrainment component and possiblyone or more auxiliary components, are provided in other embodiments.Accordingly, some embodiments disclosed herein are configured to utilizeonly entrainment components, and such embodiments can be implementedusing closed-loop or open-loop arrangements.

One or more embodiments can be illustratively configured to provide auser with an awareness of a current state and guidance towards anoptimal state, possibly with appropriate rewards for achieving and/ormaintaining the optimal state, and can provide additional or alternativefunctionality.

These and other illustrative embodiments include but are not limited tosystems, methods, apparatus, and computer program products. Theillustrative embodiments are advantageously configured to address andsolve one or more significant problems of conventional approaches, asoutlined in more detail elsewhere herein.

BRIEF DESCRIPTION OF THE FIGURES

A further understanding of the nature and advantages of particularembodiments may be realized by reference to the remaining portions ofthe specification and the drawings, in which like reference numerals areused to refer to similar components. In some instances, a sub-label isassociated with a reference numeral to denote one of multiple similarcomponents. When reference is made to a reference numeral withoutspecification to an existing sub-label, it is intended to refer to allsuch multiple similar components.

FIG. 1 is a schematic diagram illustrating a system for implementing thegeneration of music and audio, in accordance with various embodiments.

FIGS. 2A-2C illustrate user interface designs for interacting with themusic/audio and music/audio generation system, in accordance withvarious embodiments.

FIG. 3 is a schematic diagram illustrating a system for implementing thegeneration of music and interactive audio, in accordance with variousembodiments.

FIGS. 4A-4C are schematic diagrams illustrating systems for mappingmusic/audio to emotions/states, in accordance with various embodiments.

FIG. 5 is a flow diagram illustrating a method for implementing thegeneration of music/audio, in accordance with various embodiments.

FIG. 6 is a flow diagram illustrating a method for implementing a userinterface for the generation of music/audio, in accordance with variousembodiments.

FIGS. 7A-7C are flow diagrams illustrating a method for implementing auser interface for the generation of music/audio, in accordance withvarious embodiments.

FIG. 8 is a flow diagram illustrating a method for generatingmusic/audio, in accordance with various embodiments.

FIGS. 9A-9E are schematic diagrams illustrating systems for mappingstates to music, in accordance with various embodiments.

FIG. 10 is a flow diagram illustrating a method for continuouslycontrolling the generation of the music to guide a user toward a goal,in accordance with various embodiments.

FIG. 11 is a block diagram illustrating an exemplary computer or systemhardware architecture, in accordance with various embodiments.

FIG. 12 is a block diagram illustrating a networked system of computers,computing systems, or system hardware architecture, which can be used inaccordance with various embodiments.

FIGS. 13 through 24 show aspects of breathing entrainment sonificationtechniques in illustrative embodiments.

FIGS. 25 through 29 show aspects of musical communication systems andassociated applications including breathing entrainment sonification inillustrative embodiments.

FIGS. 30 through 32 show aspects of goal-driven auditory displaytechniques in illustrative embodiments.

FIG. 33 shows an example of an entrainment signal utilized in abreathing entrainment sonification system in an illustrative embodiment.

DETAILED DESCRIPTION

Some illustrative embodiments of the present disclosure relate, ingeneral, to methods, systems, and apparatuses for implementingprocedural, generative, interactive, and reactive music and audio, and,more particularly, to methods, systems, and apparatuses for generatingmusic and audio associated with a state or an emotion contained within acommunication, for a user interface for generating and controlling musicand audio characteristics associated with a state or an emotion, forgenerating audio to inform the user of a current state, for generatingaudio to give the user awareness of movement between states, and forgenerating audio to guide a user toward a desired state or goal.

While there are several services that generate music on a computer, noneof these services tailor the music and audio generated in real-time ornear real-time to an emotion or a current state of a particularuser/person or environment. Further, none of these services are drivenby an emotion or state of a user/person contained within a communicationor have an intuitive interface for a user/person to influence the audiothat allows for smooth transformations between different musical states.Additionally, none of these services seek to direct and/or guide a usertoward a desired state or goal.

Hence, there is a need for a more robust, and portable solution forgenerating music/audio, and, more particularly, for generatingmusic/audio associated with an emotion/state contained within acommunication, for a user interface for generating music/audioassociated with a state, and for generating music to guide a user towarda desired state or goal.

Illustrative embodiments of methods and systems for musicalcommunication will now be described with reference to FIGS. 1 through12. For example, some of these embodiments provide novel tools andtechniques for generating music, and, more particularly, methods,systems, and apparatuses for generating music associated with a statecontained within a communication, for a user interface for generatingmusic associated with a state, for generated music to reflect movementbetween states, and for generating music to guide a user toward adesired state. In various embodiments, a computing system might analyzea communication of a user to determine at least one state containedwithin the communication. The communication may be a sensorcommunication, a biometric/health communication, a voice communication,a numerical communication, a textual communication, a picture/videocommunication, etc. Based on the determined at least one state, thecomputing system might generate music associated with the state orcontinuously control the generation of music to guide a user toward adesired state. A user interface for generating music may also beprovided.

Accordingly, various embodiments disclosed herein provide, for example,tools and techniques for generating interactive music/audio, and, moreparticularly, methods, systems, and apparatuses for generating musicassociated with an emotion/state contained within a communication, for auser interface for generating music/audio associated with an emotion,and for generating music to guide a user toward a desired state or goalor more optimally performing in an environment.

Some methods, systems, and apparatuses disclosed herein provide meansfor musical/audio communication. In an age of digital communicationdominated by screens, text, and interactive mediums composed of staticsound files (e.g., mp4 and .wav), there is a need to express informationthrough a flexible audio feedback system. Understanding of communicationthrough real-time adaptive music and audio can be achieved bytransforming musical states which feature characteristic emotionalnuance. These emotional sound qualities and changing states may beexpressed through music/audio feature parameters associated withparticular states and interpolating between different state parameters.

The methods, systems, and apparatuses described herein provide an audioengine with generative audio/music models and interfaces that generateor synthesize music or interactive sound in real-time based on feedbackfrom one or more sensors and/or user interaction. These models canproduce procedurally generated audio featuring infinite variation andinfluence from one or more sensors and/or user interactions. Thisapproach consists of mapping detailed music content, performance andexpressive features and components to input data streams andprobabilistic structures.

Further, the methods, systems, and apparatuses described herein providepersonalized, interactive, generative music and/or interactive audiothat adapt to and convey a user's current physiological state(emotional, physical, or the like), the state of a person that the useris interacting with via text, video, voice communication, and/or thelike, the state of a technology communicating to a user, and/or thestate of an environment surrounding a user or a device. For example,instead of sending text/photographic emojis, users of these methods,systems, and apparatuses may send musical emojis to express how they arefeeling. Additionally and/or alternatively, the music/sound that isgenerated may reflect a state of a technology (e.g., an amount of power,types of applications that are open, a typing speed of the user, and/orthe like). In some cases, the music/audio may reflect a state of theenvironment (e.g., a type of weather, temperature of a room, and/or thelike).

The methods, systems, and apparatuses described herein may be used toprovide a soundtrack to user activities including goal-oriented tasks,communications (e.g., text, voice, video, and/or the like), userapplications (e.g., games, social media applications, and/or the like),video games, a user's environment, and/or the like. Real-time datastreams input into the generative music model may come from one or moresensors, user interaction with a user interface, computer software, acomputer program, a virtual world, and/or the like. Additionally and/oralternatively, by interpreting the body language, tone of voice, brainwaves, pulse of a user and/or person a user is interacting with and/orby detecting an environment within a room, a surrounding area, and/orthe like, the methods, systems, and apparatuses described herein maygenerate instant audio feedback on psychological state of the userand/or a person interacting with the user, and/or an environment of auser or a person interacting with the user in the form of complex,sounds abstracting natural human expression. Thus, through the methods,systems, and apparatuses described herein, a user's mind, voice, body,environment, and/or the like may be used as input data driving themusical instrument.

Physiological states and/or environmental states are temporal in nature,therefore they can most be conveyed highly accurately through evolvingmusic or sound. Thus, the music and audio that is generated, by methods,systems, and apparatuses described herein, is not static and maytransition, transform or diverge over time as the user's state orenvironment changes.

These methods, systems, and apparatuses could also be embraced as anaccessibility tool, facilitating blind and autistic communication.Further, these methods, systems, and apparatuses could be used insituations where text/photograph emojis fail, such as in accessibilitydevices for the blind or with individuals with autism. Additionally,these methods, systems, and apparatuses could be used for therapysessions to make a patient feel at ease and to provide feedback to thetherapist about the emotional state of the patient or be used for soundtherapy for the patient.

Additionally and/or alternatively, the methods, systems, and apparatusesdisclosed determine different emotions contained within a communication(e.g., a biometric/health communication, a voice communication, atextual communication, a picture communication, a video communication, ahaptic communication and/or the like). Based on the determined emotions,the methods, systems, and apparatuses may generate and play music and/orother audio content associated with at least one determined emotion. Thegenerated music and/or other audio content would provide a soundtrack tothe communication and add emotional nuance to the communication throughmusic to more effectively express how a person is feeling.

In other embodiments disclosed, an intuitive user interface is providedto a user such that a user may generate music associated with differentemotions and smoothly transition between generating music associatedwith different emotions. The user interface provides for smoother andless audible jarring of music when transitioning between musicassociated with different emotions, and/or the like. The unique input ofeach user will cause the music that is generated to uniquely vary insound and expression.

Additionally and/or alternatively, these methods, systems, andapparatuses may be used as assistive sound guides for goal-basedactivities. Continuous control over musical content and expression canallow for morphing between different musical states to convey to theuser the direction they are moving in from their initial state. Thechanging characteristics in the audio can indicate to the user thedirection moved in from the initial state, if they are closer or fatheraway from their goal, how quickly they are moving in either direction.These methods, systems, and apparatus may be used to generatecontinuously evolving and/or changing sound characteristics over time todirect a user toward a desired state or goal. In a non-limiting example,various embodiments may be configured to exercise continuous controlover musical/sound characteristics based on input from one or moresensors (e.g., one or more biometric/health sensors, motion sensors,distance sensors, GPS sensors, touch screen position, pressure sensorand/or the like). In some embodiments, the musical characteristics mayalso evolve or change to act as a guide to guide a user to a desiredlocation (e.g., conveying a person's current position and/or thedirection he or she is moving).

In several non-limiting examples, the musical/sound characteristics mayact to guide a user from a negative state to a positive state, from astressed state to a relaxed state, from a non-meditative state to ameditative state, from an unfocused state to a focused state, from arestful state to an exercise state, from an exercise state to a restfulstate, from a first location to a second location, and/or the like.Additionally and/or alternatively, the music may guide a user throughincreasing/decreasing an intensity of a workout, increasing/decreasing abreathing pattern, increasing/decreasing a heartrate,increasing/decreasing a number of steps per minute, and/or the like. Inother embodiments, the music may act as a musical guide/navigator forthe visually impaired or to enhance the experience of a visual task suchas navigation in the real or virtual environment. In yet otherembodiments, the evolving music may guide a user to better postureand/or balance.

The following detailed description illustrates a few exemplaryembodiments in further detail to enable one of skill in the art topractice such embodiments. The described examples are provided forillustrative purposes and are not intended to limit the scope of theinvention.

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the described embodiments. It will be apparent to oneskilled in the art, however, that other embodiments of the presentinvention may be practiced without some of these specific details. Inother instances, certain structures and devices are shown in blockdiagram form. Several embodiments are described herein, and whilevarious features are ascribed to different embodiments, it should beappreciated that the features described with respect to one embodimentmay be incorporated with other embodiments as well. By the same token,however, no single feature or features of any described embodimentshould be considered essential to every embodiment of the invention, asother embodiments of the invention may omit such features.

Unless otherwise indicated, all numbers used herein to expressquantities, dimensions, and so forth used should be understood as beingmodified in all instances by the term “about.” In this application, theuse of the singular includes the plural unless specifically statedotherwise, and use of the terms “and” and “or” means “and/or” unlessotherwise indicated. Moreover, the use of the term “including,” as wellas other forms, such as “includes” and “included,” should be considerednon-exclusive. Also, terms such as “element” or “component” encompassboth elements and components comprising one unit and elements andcomponents that comprise more than one unit, unless specifically statedotherwise.

Various embodiments described herein, while embodying (in some cases)software products, computer-performed methods, and/or computer systems,represent tangible, concrete improvements to existing technologicalareas, including, without limitation, user communication technology,music content generation technology, music content navigation orselection technology, user interface technology, audio playbacktechnology, and/or the like. In other aspects, certain embodiments, canimprove the functioning of user equipment or systems themselves (e.g.,music players, music streaming or downloading systems, audio playbackdevices, etc.), for example, by, analyzing, with a computing system, acommunication to determine at least one emotion or state containedwithin the communication, embodiments can determine a particular statethat a user/person or technology is experiencing and generate musicassociated with the at least one state contained within thecommunication. Additionally and/or alternatively, a user interface maybe provided that facilitates the generation of music and transitionsbetween music associated with different states to convey to the user thecurrent state musically. In particular, to the extent any abstractconcepts are present in the various embodiments, those concepts can beimplemented as described herein by devices, software, systems, andmethods that involve specific novel functionality (e.g., steps oroperations), such as, analyzing a communication to determine at leastone emotion or state contained within the communication and generatingmusic associated with the at least one emotion contained within thecommunication and utilizing a user interface to generate musicassociated with a particular emotion and to smoothly transition betweenmusic associated with different states, and/or the like, to name a fewexamples, that extend beyond mere conventional computer processingoperations. These functionalities can produce tangible results outsideof the implementation of a computer system, including, merely by way ofexample, optimized presentation of audio content (e.g. music associatedwith an emotion) to a user, generation of audio content (e.g. musicassociated with an emotion or state), and transitioning of audio content(e.g. music associated with an emotion or state). The presentation ofaudio content allows users to provide a soundtrack to theircommunications or function as the communication itself. Thegeneration/transitioning of audio content provides for smoother and lessaudibly jarring changing of audio content, and/or the like, at leastsome of which may be observed or measured by customers.

Various modifications and additions can be made to the embodimentsdiscussed without departing from the scope of the invention. Forexample, while the embodiments described above refer to particularfeatures, the scope of this invention also includes embodiments havingdifferent combination of features and embodiments that do not includeall of the above described features.

We now turn to the embodiments as illustrated by the drawings. FIGS.1-12 illustrate some of the features of the method, system, andapparatus for generating music (and/or other audio/sound content), and,more particularly, to methods, systems, and apparatuses for generatingmusic (and/or other audio/sound content) associated with an emotioncontained within a communication and for a user interface for generatingmusic associated with an emotion or state, as referred to above.Although the specification generally refers to generating music, itshould be noted that other interactive audio/sound content (e.g.,reactive sound, reactive audio, and/or the like) may be generated usingthe methods, systems, and apparatuses described below. The methods,systems, and apparatuses illustrated by FIGS. 1-12 refer to examples ofdifferent embodiments that include various components and steps, whichcan be considered alternatives or which can be used in conjunction withone another in the various embodiments. The description of theillustrated methods, systems, and apparatuses shown in FIGS. 1-12 isprovided for purposes of illustration and should not be considered tolimit the scope of the different embodiments.

With reference to the figures, FIG. 1 is a schematic diagramillustrating a system 100 for generating music, in accordance withvarious embodiments. Although the generation of music is often referredto throughout the specification, a person of ordinary skill in the artcan understand that similar methods that generate music may be used togenerate sound and/or audio.

In the non-limiting embodiment of FIG. 1, system 100 might comprise oneor more user devices 105 (also referred to as “computing system 105”)and a data store or database 110 a that is local to the one or more userdevices 105. In some cases, the database 110 a might be external, yetcommunicatively coupled, to the one or more user devices 105. In othercases, the database 110 might be integrated within a user device 105.User device 105 might comprise a display 105 a. The display 105 a may bea touchscreen display that is configured to receive tactile input from auser or a display that is configured to receive input from a mouse. Thedatabase system may apply some preprocessing to the data before it isfed into the music system as an input. Additionally and/oralternatively, system 100 may further comprise one or more input devices115 and one or more audio playback devices 120 a-120 n (collectively,“audio playback devices 120” or “speakers 120” or the like), and/or thelike.

Each of the one or more user devices 105, the one or more input devices115, and/or the one or more audio playback devices 120, might becommunicatively coupled to each other, either via wireless connectionand/or via wired connection. Additionally and/or alternatively, each ofthe one or more input devices 115, and/or the one or more audio playbackdevices 120 might be integrated within user device 105.

The one or more user devices 105 might each be configured to receiveuser input from a user. The user input may be received through touchinput from the user, through the use of a mouse, and/or the like. Theone or more user devices may further be configured to receivecommunications from data streams (e.g., text communications, voicecommunications, Internet of Things (“IoT” e.g., smart home appliance)communications, video communications, biometric/health or physiologicalsensor communications, and/or the like), according to some embodiments.In some cases, the user devices 105 might include, without limitation, adesktop computer, a television, a tablet computer, a laptop computer, avideo game console, a smart phone, an e-reader, a smart watch, aportable fitness tracker, an electroencephalography (“EEG”) device,medical equipment, fitness gym equipment, a virtual reality (“VR”)device, an augmented reality (“AR”) device, and/or the like. The one ormore user devices 105 may further be configured to receivecommunications from one or more input devices 115. The one or more inputdevices 115 may include, without limitation, a tablet computer that hasbeen paired, synced, or synchronized with the user device 105, a laptopcomputer that has been paired, synced, or synchronized with the userdevice 105, a smart phone that has been paired, synced, or synchronizedwith the user device 105, a sensor that has been paired, synced, orsynchronized with the user device 105, a biometric/health orphysiological sensor that has been paired, synced, or synchronized withthe user device 105, a fitness tracker that has been paired, synced, orsynchronized with the user device 105, an EEG device that has beenpaired, synced, or synchronized with the user device 105, a virtualreality (“VR”) device that has been paired, synced, or synchronized withthe user device 105, an augmented reality (“AR”) device that has beenpaired, synced, or synchronized with the user device 105, a camera thathas been paired, synced, or synchronized with the user device 105, afacial recognition sensor that has been paired, synced, or synchronizedwith the user device 105, a distance sensor that has been paired,synced, or synchronized with the user device 105, a motion sensor thathas been paired, synced, or synchronized with the user device 105, amovement sensor that has been paired, synced, or synchronized with theuser device 105, a skin conductance sensor that has been paired, synced,or synchronized with the user device 105, a speed or velocity sensorthat has been paired, synced, or synchronized with the user device 105,an air movement sensor that has been paired, synced, or synchronizedwith the user device 105, a pressure sensor that has been paired,synced, or synchronized with the user device 105, an accelerometer thathas been paired, synced, or synchronized with the user device 105, agyroscope sensor that has been paired, synced, or synchronized with theuser device 105, an IoT sensor that has been paired, synced, orsynchronized with the user device 105, a temperature sensor that hasbeen paired, synced, or synchronized with the user device 105, a weathersensor that has been paired, synced, or synchronized with the userdevice 105, a humidity sensor that has been paired, synced, orsynchronized with the user device 105, one or more security sensors thathas been paired, synced, or synchronized with the user device 105, asmart home interface device (e.g., echo, etc.) that has been paired,synced, or synchronized with the user device 105, and/or the like and/orcombinations of the like.

Each of the one or more user devices 105 and/or input devices 115 may belocated with a user, at a customer premises, in a home, in a car, at agym, at a wellness centers, in a medical center, in a physical therapycenter, at a hotel, at a retail store, and/or the like.

In some embodiments, the computing system 105 might comprise one of aprocessor within the user device 105 running a software application(“app”), a processor within the input device 115 running an app, aprocessor within one of the audio playback devices, and/or the like. Insome embodiments, the audio playback devices 120 might each include,without limitation, one or more speakers external to but communicativelycoupled to the user device 105 and/or input device 115, one of one ormore speakers integrated within the user device 105 and/or input device115, one or more headphones, one or more earbuds, one or more soundbars, one or more wireless speakers, or one or more stereo speakers,and/or the like.

System 100 might further comprise one or more music content sources orservers 125 or music generation sources or servers 125 and correspondingdatabases 130 that might communicatively couple to the computing system105 via one or more networks 140 (and in some cases, via one or moretelecommunications relay systems 150, which might include, withoutlimitation, one or more wireless network interfaces (e.g., wirelessmodems, wireless access points, and the like, one or more towers, one ormore satellites, and/or the like). The lightning bolt symbols are usedto denote wireless communications between the one or moretelecommunications relay systems 150 and each of at least one of theuser devices 105, between the telecommunications relay systems 150 andeach of at least one of the input devices 115, between the one or moreuser devices 105 and each of at least one of the input devices 115,between the user devices and each of the one or more audio playbackdevices 120 a-120 n, between the input devices 115 and each of at leastone of the one or more audio playback devices 120 a-120 n, and/or thelike.

At least one of computing system 105, input device 115, or at least oneaudio playback device 120 might receive a communication and/or userinput, the user communication and/or user input may contain and/orindicate at least one state of a user, a person other than a user, astate of an environment, a state of a digital book, a state of a videogame, and/or the like. A state of the user and/or a person other than auser might correspond to at least one of an emotion of a user and/or aperson other than a user, a feeling of a user and/or a person other thana user, a location of the user and/or a person other than a user, aphysical position of a user and/or a person other than a user, a levelof activity of a user and/or a person other than a user, an action of auser and/or a person other than a user, and/or the like. A state of anenvironment might correspond to at least one of a weather situation(e.g., sunny, rainy, etc.), a temperature of an area, an amount ofhumidity, an amount of light, a time of day, a time of year, and/or thelike. A state of a digital book might be at least one of a state of oneor more characters in a book, a scene of a book (e.g., action,suspenseful, etc.), and/or the like. A state of a video game might be atleast one of a state of one or more characters in a book, a scene of abook (e.g., action, suspenseful, etc.), and/or the like. A processor ofthe computing system 105, input device 115, or at least one audioplayback device 120 may analyze the user and/or communication todetermine at least one state indicated by the user input and/or at leastone state contained within the communication.

In some embodiments, the communication that is analyzed to determine atleast one state of a user, a person other than a user, an environment,etc. may be at least one of a sensor communication, an IoT sensorcommunication, a biometric/health communication, a movement-basedgesture communication, a voice communication, a textual communication, aphotographic communication, a video communication, a virtual reality(“VR”) communication, an augmented reality (“AR”) communication, anumerical communication, a vehicular communication, and/or the like. Thecomputing system 105, input device 115, and/or at least one audioplayback device 120 may receive input from the one or morecommunications periodically (e.g., every second, every minute, every fewseconds, every few minutes and/or the like).

A sensor communication may contain feedback from one or more sensorsincluding, but not limited to, one or more GPS sensors, one or moredistance sensors, one or more motion sensors, one or more movementsensors, one or more speed or velocity sensors, one or moreaccelerometer sensors, one or more gyroscope sensors, one or morebiometric/health sensors, one or more facial recognition sensors, one ormore cameras, one or more weather sensors, one or more temperaturesensors, one or more ambient light sensors, one or more humiditysensors, one or more audio sensors, and/or the like. Based on input fromthe one or more sensors, the computing system 105, input device 115,and/or at least one audio playback device 120 may determine a state thata person is experiencing or a state of the environment.

An IoT sensor communication may contain feedback from one or more IoTsensors contained within a home. The IoT communication may be sent byone or more smart home devices (e.g., echo, google home, etc.). Forexample, the one or more IoT sensors might include one of one or morethermometers in one or more rooms, one or more infrared (“IR”)thermometers aimed at one or more positions in the one or more rooms,one or more air flow sensors in the one or more rooms, one or more airflow sensors in air ducts directed toward the one or more rooms, one ormore indoor solar light sensors, one or more outdoor solar lightsensors, one or more outdoor wind sensors, one or more neighborhoodweather station sensors, one or more regional weather station sensors,one or more motion detectors detecting presence of people or animals inat least one of the one or more rooms or outside the customer premises,one or more humidity sensors in the one or more rooms, one or more smokedetectors detecting smoke in the one or more rooms, one or more gasdetection sensors detecting gas in the one or more rooms, one or morebiometric sensors identifying at least one person, or one or more healthdevice with sensors detecting health information for at least oneperson, and/or the like. Based on input from the one or more sensors,the computing system 105, input device 115, and/or at least one audioplayback device 120 may determine a state that a person is experiencingor a state of the environment. The music may be adapted to change inreal-time as the state of the user, other person, or environment changesbased on feedback from the one or more sensors contained in the IoTcommunications.

A biometric/health communication may contain biometric/health feedbackfrom at least one of a medical device, smart phone, a smart watch, afitness tracker, an EEG device, and/or the like. The biometric/healthfeedback may include at least one of a heart rate, a HRV, a breathingrate, a blood pressure, a stress level, a measure of electrical activitywithin a brain, pupil dilation, skin conductivity, and/or the like.Based on the at least one of a heart rate, a blood pressure, a stresslevel, a measure of electrical activity within a brain, and/or the like,the computing system 105, input device 115, and/or at least one audioplayback device 120 may determine a state that a person or anenvironment is experiencing.

In a non-limiting example, if a heart rate is elevated, then thecomputing system 105, input device 115, and/or at least one audioplayback device 120 may determine that a person is stressed. The musicthat the computing system 105, input device 115, and/or at least oneaudio playback device 120 generates may be louder and have a fasterrhythm to reflect that a person is feeling stressed. Alternatively, thesystem may play music to compliment a person's state and play calmingmusic when the person is stressed to guide a user from a negative stateinto a more positive state. Additionally and/or alternatively, in anon-limiting example, if a heart rate is low and/or skin conductivity orpupil dilation is low then the computing system 105, input device 115,and/or at least one audio playback device 120 may determine that aperson is calm. The music that the computing system 105, input device115, and/or at least one audio playback device 120 generates may reflectthe person's calm state and/or the music that is generated may bedesigned to cause a person to become excited. The music may be adaptedto change in real-time as the state of the user and/or other personchanges in the biometric/health communication. Preset parameters withinthe music system define different states. Interpolation of theparameters defining each state allows for continuous morphing of themusic between states.

Additionally and/or alternatively, multiple biometric communications maybe received from multiple users. The music generated from thecommunications may be used to reflect how one or more users are doing ina competitive activity (e.g., a swim race, a track race, virtual raceetc.). Additionally and/or alternatively, the music that is generatedmay be used to reflect the compatibility/connectiveness of a team.

A voice communication may be obtained from at least one of a phone call,voice input, microphone input, and/or the like. The computing system105, input device 115, and/or at least one audio playback device 120 mayparse the words contained within the voice communication to determineone or more particular states that at least one person is experiencing.Additionally and/or alternatively, the computing system 105, inputdevice 115, and/or at least one audio playback device 120 may determinea tone of voice used in the voice communication to determine at leastone state that a person is experiencing. For example, a higher, loudertone of voice might indicate happiness while a lower, quieter tone ofvoice might indicate sadness. The music may be adapted to change inreal-time as the state of the user and/or other person changes in thevoice communication.

A textual communication may be obtained from at least one of a textmessage, an email message, an instant message, a webpage, an e-book,and/or the like. The computing system 105, input device 115, and/or atleast one audio playback device 120 may parse the words contained withinthe textual communication to determine one or more particular statesthat at least one person and/or character is experiencing. Computingsystem 105, input device 115, and/or at least one audio playback device120 may perform machine translation on the text. The computing system105, input device 115, and/or at least one audio playback device 120 mayfurther determine whether one or more emojis are contained within thetextual communication and determine one or more states associated withthe one or more emojis contained within the textual communication. Themusic may be adapted to change in real-time as the state of the userand/or other person changes in the textual communication.

A photographic communication may be obtained from at least one of aphotograph, video, and/or the like. The computing system 105, inputdevice 115, and/or at least one audio playback device 120 may analyzethe facial expression, body language, and/or the like of one or morepersons in the photograph to determine one or more particular statesthat the at least one person or character is experiencing. The computingsystem 105, input device 115, and/or at least one audio playback device120 may analyze the facial expression, body language, and/or the like ofone or more persons and/or characters in the video to determine one ormore particular states that the at least one person/character isexperiencing. The computing system 105, input device 115, and/or atleast one audio playback device 120 may also parse the dialogue of theat least one person in the video or determine a tone of voice of the atleast one person in the video to determine one or more particular statesthat the at least one person/character is experiencing. The music may beadapted to change over time as the state of the user and/or other personchanges in the photographic communication.

A VR communication and/or AR communication may be obtained from VR/ARdevices (e.g., cell phones, tablets, headsets, glasses, goggles, lenses,and/or the like) or be a parameter built into the game determined by thecontext within the game. The computing system 105, input device 115,and/or at least one audio playback device 120 may analyze the facialexpression, body language, and/or the like of the user of the AR/VRdevice, one or more persons interacting/communicating with the user ofthe AR/VR device, and/or one or more characters displayed by the AR/VRdevice. The computing system 105, input device 115, and/or at least oneaudio playback device 120 may also parse the dialogue or determine atone of voice of the user, one or more persons interacting with the userof the AR/VR device, and/or one or more characters displayed by theAR/VR device. Additionally and/or alternatively, the computing system105, input device 115, and/or at least one audio playback device 120 maydetect how a person is performing in a game and generate music based onthe person's performance. In non-limiting examples, if the user is doingwell, then the music that is generated may be more upbeat, if the useris in an action scene, the music that may be generated may be moreintense, if the character is performing poorly, then the music generatedmay become more tense, if the user's character dies in the game, thenthe music that is generated may contain lower tones, and/or the like.The music may be adapted to change continuously in real-time as thestate of the user and/or other person changes in the VR/ARcommunication.

A numerical communication may be obtained from one or more tradingapplications, option prices, profit/loss calculations, riskcalculations, etc. The audio may change continuously to convey real-timeprice changes, risk changes, etc.

A vehicular communication may be obtained from one or more vehicularcomponents (e.g., one or more location sensors, one or more motionsensors, one or more speed sensors, or more velocity sensors, one ormore GPS, one or more acceleration sensors, one or more pressuresensors, one or more force/impact sensors, one or more steering sensors,one or more autonomous vehicle controls, one or more self-drivingcontrols, one or more position sensors, one or more other vehicularsensors, a horn, and/or the like). The music or audio may be adapted tochange continuously in real-time as the state/action of the user, otherperson, environment of surrounding a vehicle, and/or the vehicle changesin the vehicular communication. For example, based on the input from thevehicular component, the system may determine the driver or passenger isdistracted or stressed. Music or audio may then be generated to help thedriver or passenger focus or relax. Additionally, the audio generatedcan be used to enhance the communication of a horn, where differentstates can convey information to another vehicle about why the hornsignal was pressed by the driver or passenger.

Additionally and/or alternatively, the computing system 105, inputdevice 115, and/or at least one audio playback device 120 may receive anindication from a user on a display of the computing system 105, theinput device 115, and/or the at least one audio playback device 120. Theindication may be received via tactile input or input via a mouse. Eachstate may be mapped to a position in a circular pattern (shown in FIG.2), an XY coordinate system (shown in FIG. 9D), an XYZ coordinatesystem, and/or the like.

In some embodiments, the music/audio that is generated may be based on acombination of two or more of one or more sensor inputs, one or morecommunications, and/or one or more user indications.

Based on the at least one determined state indicated by one or moresensor inputs, contained within the communication, and/or indicated bythe user input, the processor of the computing system 105, input device115, and/or at least one audio playback device 120 may autonomouslydetermine one or more first characteristics of a plurality ofcharacteristics of music/audio associated with the determined at leastone state indicated by one or more sensor inputs, contained within thecommunication, and/or indicated by the user. Additionally and/oralternatively, the computing system 105, input device 115, and/or atleast one audio playback device 120 may access database 110 a and/ormusic source 125 to determine one or more first characteristics of aplurality of characteristics of music associated with the determined atleast one state contained within the communication and/or indicated bythe user. The one or more first characteristics of the plurality ofcharacteristics of music may include at least one of a note selection, anote pattern, an envelope, a harmony or combination of notes, anorchestration, a timbre quality, a filter, a speed, a rhythm, and/or avolume associated with the first state indicated by the communication.

The communication and/or indication may further indicate an age and/orsex of a user. The one or more first characteristics of a plurality ofcharacteristics of music may further be associated with the at least oneof the age or the sex contained within the communication and/orindicated by the user input. The music that is generated may furtherhave one or more characteristics of the plurality of characteristicsassociated with the at least one of the age or the sex indicated by thecommunication and/or user input. In a non-limiting example, peopleassociated with an older age demographic may prefer classical musicrhythms and orchestrations while people associated with a younger agedemographic may prefer hip-hop music rhythms and orchestrations. Thedemographics may determine initial position of the music system andshape the directions, (states or presets) that the system will move to.

Additionally and/or alternatively, a user may be able to indicatecharacteristics of music that the user prefers to have. For example, auser may prefer to listen to hip hop music. Computing system 105, inputdevices 115, and/or playback devices 120 may receive this input(indicating that a user prefers hip hop music) and generate music havinghip hop characteristics (e.g., featuring sub bass, louder lowfrequencies, characteristic rhythmic attributes, and/or the like).

A user may explicitly indicate a particular preference for certain typesof music and/or computing system 105 may determine types of music a userprefers. For example, computing system 105 may monitor the type of musica user buys or listens to on an application (e.g., a music application).Based on the determined music a user buys and/or listens to on theapplication, the computing system 105 may generate music that hassimilar characteristics. For example, a user may buy and listen tomostly classical music, the computing system 105 may then generate musichaving classical characteristics such as orchestra sounding instrumentsso that the music that is generated in appealing to the user.

Additionally and/or alternatively, instead of detecting states,computing system 105 may detect different types of elements and/orconcepts. In a non-limiting example, computing system 105 may detectdifferent types of scenes (e.g., action scene, adventure scene,landscape scenes, traffic scenes, and/or the like) of a book, videogame, movie, and/or the like. The music that is generated may furtherhave one or more characteristics of the plurality of characteristicsassociated with the at least one of the type of scene indicated by thecommunication and/or user input. In a non-limiting example, a particularpart of a book may have a sword fight and the music that is generatedmay have a faster rhythm and louder volume to reflect that the scene isan action scene. Additionally and/or alternatively, computing system 105may detect a state and/or action (e.g., running, walking, biking,sitting down, swimming, driving, etc.) of the user. For example, thecomputing system 105 may generate faster music when it determines aperson is running and generate slower music when it determines a personis sitting down. Alternatively, a car may be stuck in traffic and themusic that is generated might be designed to calm a driver.Alternatively, a driver may be falling asleep and the music generatedmight be designed to wake up the driver.

In additional embodiments, computing system 105 may detect a type ofapplication that is using the music generation system. The music that isgenerated may be adjusted to fit the application, game, sensor system,and/or the like. For example, if the music generation system is beingused to supplement a voice conversation, then the music that isgenerated may contain lower tones/volumes. If the music generationsystem is being used to supplement information received from a fitnesstracker, then the music that is generated may contain highertones/volumes or stronger rhythmic attributes. If the music isfunctioning as background sound in a VR/AR game, it may not only portraypsychological and physical state of the player but also convey player'sperformance in the game.

Based on the determination of the one or more first characteristics of aplurality of characteristics of music associated with the computingsystem 105, input device 115, and/or at least one audio playback device120 may generate music having the one or more first characteristics ofthe plurality of characteristics associated with the at least onestate/element/concept contained within the communication and/orindicated by the user. The music may change as the input(s) from the oneor more communications change. The music may change periodically (e.g.,every second, every minute, every few seconds, every few minutes). Thegenerated music may further contain one or more first characteristicsassociated with the at least one of the age or the sex indicated by thecommunication and/or user input. Additionally and/or alternatively, thegenerated music may further have one or more characteristics indicatedby a preference of the user.

The music may be generated using one or more of frequency modulation,additive synthesis, subtractive synthesis, wave table synthesis,granular synthesis or sample-based synthesis and/or the like. The musicmay be generated from input from one or more communications (e.g., asensor communication, an IoT sensor communication, a biometric/healthcommunication, a movement-based gesture communication, a voicecommunication, a textual communication, a photographic communication, avideo communication, a virtual reality (“VR”) communication, anaugmented reality (“AR”) communication, a tactile communication, and/orthe like). The music generated from two or more communications may besynthesized and harmonized together.

Additionally and/or alternatively, the music that is generated may bedesigned to have an opposite effect on the user listening to the music,complimenting their current state. For example, if the computing system105, input device 115, and/or at least one audio playback device 120determines that a user is sad based on the state detected within thecommunication, then the music that is generated may have one or morecharacteristics associated with the emotion “Happy” to uplift the user'sspirits.

Each emotion, age, sex, and/or preference may have a customizedalgorithm for generating music that reflects the determined emotion,element/concept, scene, age, sex, and/or preference of a person. Thesealgorithms may be inspired by soundtrack clichés, like those used inHollywood films and/or used by composers. These algorithms may beinspired by associations of musical attributes to emotion perceived inmusic from psychoacoustics research. These algorithms may be containedin database 110 a, music source 125, and/or database 130.

The music that is generated may be digital instruments which sound likereal instruments (e.g., violins, violas, flutes, drums, etc.).Additionally and/or alternatively, the music that is generated may notbe limited to instrumental music, but rather, synthesized electronicinstruments, that may compose the sound generated to convey a humanemotion. The model can be, for instance, generalized to include vocalsynthesis and/or imitate human or animal sounds (e.g., birds, whales,and/or the like). Synthesized vocalizations may imitate natural humanexpressions and responses with associated emotions.

The generated music may be configured to evolve over time and transitionbetween different states as the user's state changes, the one or morepersons' state changes, and/or the state of the environment changes.Additionally and/or alternatively, the music may be configured to guidea user toward a desired state by continuously adapting based on at leastone of the one or more sensor inputs, the one or more communications,and/or one or more indications by a user.

In yet another non-limiting example, the computing system 105, inputdevice 115, and/or at least one audio playback device 120 may exercisecontinuous control over one or more musical characteristics to cause theone or more musical characteristics to evolve or change over time andguide a user from one state to another state. For example, based onfeedback received from the one or more communications, the computingsystem 105, input device 115, and/or at least one audio playback device120 may continuously control the one or more musical characteristics toguide a user from a first state to a second desired state. This processwill be described more with respect to FIGS. 9 and 10, below.

The music that is generated by computing system 105 may further havehuman-like embellishments. Human performers have a natural imprecisionwhich must be explicitly accounted for in computer generated music. Todo this, irregular micro-fluctuations are added to the timing of noteonsets. This kind of small random signal bias is often referred to as“timing jitter.” As a result, quantized notes are gently un-quantized toprovide a more pleasing and human-sounding musical aesthetic. Timingjitter provides subtle rhythmic variation that add nuance to the staticnote patterns.

Similar to jittered timing offsets (“timing jitter”), “frequency jitter”is utilized to modulate the frequency (pitch) of the generated music.Depending on the duration and articulation of the note, frequency jitterparameters will change. For instance, long sustained notes will besubject to more evolved jitter (gradual drift), a technique to addwarmth; while shorter, more percussive notes will have little to nojitter.

Jitter may also be mapped to a number of other parameters in charge ofproducing timbre or the sound qualities of the notes. This is referredto as “timbral jitter.” These parameters exist due in part to thereal-time audio synthesis engine, which allow dynamic control andmodulation of a sound over time via digital signal processing.

The generated music associated with the at least one determined statemay be played through one or more playback devices 120.

In a non-limiting example, if the computing system 105, input device115, and/or at least one audio playback device 120 determines that acommunication and/or indication contains the state “Happy,” the musicthat is generated may contain higher notes and a faster rhythm. If thecomputing system computing system 105, input device 115, and/or at leastone audio playback device 120 determines that an indication and/orcommunication contains the state “Sad,” the music that is generated maycontain lower notes and a slower rhythm. Additionally and/oralternatively, if the computing system 105, input device 115, and/or atleast one audio playback device 120 determines that a communicationcontains an environmental state of “Sunny” the music that is generatedmay contain higher notes and a faster rhythm.

The computing system 105 may detect that more than one state iscontained within the communication and/or indicated by a user. If thecomputing system 105 detects at least two states, then the computingsystem may simultaneously generate, play, and/or harmonize the musicthat is associated with the at least two states. Additionally and/oralternatively, if the computing system detects at least two states, thenthe computing system may determine an order to play music associatedwith each of the at least two states and smoothly transition betweenplaying music associated with each of the at least two states. Dependingon the number of emotions contained within each indication and/orcommunication, the computing system 105 may determine an order ofsubsets of two or more states to simultaneously generate, play, and/orharmonize and smoothly transition between playing music associated witheach subset of two or more states.

If the music associated with the at least two states are played in aparticular order, the music associated with each state may be played fora predetermined amount of time (e.g., the music/sound may be playeduntil a note is finished playing, until a sequence has ended, and/or thelike) before transitioning. Additionally and/or alternatively, the musicassociated with each state may be transitioned based on a trackedposition of a user in a communication (e.g., a book or text) or based onwhich person is talking in a phone call. Computing system 105 mayfurther use a fitness tracker, an EEG device, a video communication,and/or a voice communication to track how a user's state or anotherperson's state is changing and transition to different music based on adetermined change in a user's state and/or another person's state inreal-time or near real-time. In order to transition the generated musicin real-time or near real-time, the computing system may introduceinterpolation in different areas which enables gradual and/or abrupttransition between musical parameters associated with differentemotions. There may be a mixture of interpolation including smoothedcontinuous control (which may feature a smoothing filter) and beatquantized transitioning. Interpolated aspects of music may includeharmonic, timbre, orchestration and rhythmic content in order totransition smoothly from one musical aspect or phrase to another.

In a non-limiting example, the communication may be a book, or digitalmedium with a narrative and computing system 105 may track where aparticular user is in the narrative. Based on one or more statescontained within a particular part of the narrative, the computingsystem may generate music associated with those states. The musicassociated with these states may be harmonized and/or played in theorder that each state appears on the page of book. This creates acustomized soundtrack associated with the book and adds an additionalelement beyond merely reading the text or viewing video content indigital medium.

In an additional non-limiting example, a calling party may be having aphone conversation with called party and the computing system 105 maydetermine a particular state associated with both the calling party andthe called party. The computing system may play music associated witheach state simultaneously or play music associated with the particularperson when that particular person is talking. The music may be playedat a lower volume so that the music adds additional elements to theconversation without drowning out the conversation between the callingparty and the called party.

In an additional non-limiting example, a fitness tracker may be used totrack a user as the user is transitioning through a work out. Thefitness tracker may detect that a user is cooling down from a workoutand transition the music that is generated from excited/vigorous tocalm/soothing.

In an additional non-limited example, a fitness tracker, handheld deviceor exercise machine may be used to track the cyclic nature of a workout. The tempo and rhythmic pattern of the music generated will adapt tothe cyclic activity in the workout.

In an additional example, a movement detector from a handheld device maydetect the amount of movement during a work-out. The amount of movementmay be used to increase the energy in the music generated.

Each state may be associated with an icon (e.g., a text icon and/oremoji icon) and each icon may be mapped to a circular pattern/wheel 205,shown in FIG. 2. Additionally and/or alternatively, instead of icons,each state may be mapped to a position on a circular pattern/wheel 205.FIGS. 2A-2C (collectively, FIG. 2) illustrate user interface designs 200(also referred to as user interface 200) for interacting with the musicgeneration system (as described with respect to FIGS. 1 and 3-9), inaccordance with various embodiments.

The circular pattern/wheel 205 is not limited to only being a circle.The circular pattern could be any pattern (e.g., an oval, triangle,square, rectangle, a two-dimensional graph (e.g., an XY graph, etc.) ona two-dimensional plane. Additionally and/or alternatively, the circularpattern/wheel 205 may be a three-dimensional shape or graph. A person ofordinary skill in the art would understand that any pattern may act in asimilar manner as the circular pattern 205 described below. Atwo-dimensional graph is described in more detail below with respect toFIG. 9D. The two-dimensional graph 900 d shown in FIG. 9D may be used ina similar manner as user interface 200.

According to some embodiments, the user interface 200 may display statesdetected from a communication. A user may then determine from thedisplayed states what type of music is being generated. The userinterface 200 may display only those states detected in thecommunication. Alternatively, the user interface 200 may display, aplurality of icons 210 and/or positions associated with particularstates mapped to a circular pattern 205 and the icons and/or positionsassociated with the states detected in the communication may behighlighted or bolded to stand out from the rest of the icons/positionson the circular pattern 205.

Additionally and/or alternatively, the circular pattern 205 may displayicons associated with elements/concepts other than states. For example,the icons 210 may be associated with scenes (e.g., action scenes, horrorscenes, romance scenes, and/or the like) from a movie, book, video game,and/or the like. The music that is generated may then have musicalcharacteristics associated with the scenes (e.g., action scenes, horrorscenes, romance scenes, and/or the like). Additionally and/oralternatively, the icons 210 may be associated with an action of theuser. For example, different sensors and/or input devices may detectand/or a user may input whether a person is running, walking, biking,sitting down, etc. and the generated sounds may reflect the action ofthe user. In a non-limiting example, the music that is generated, whenthe user is sitting down, may be slower than the music that is generatedwhen the user is running.

Additionally and/or alternatively, according to some embodiments (shownin FIG. 2A), a plurality of icons 210 and/or positions associated withparticular states and/or other elements/concepts may be mapped to acircular pattern 205 and displayed to a user on user interface 200. Theicons may be text icons and/or emojis. Each icon 210 and/or position maybe color coded based on the particular state and/or otherelement/concept it represents. Additionally and/or alternatively, aparticular state and/or other element/concept may be represented by anemoji associated with that particular state.

The circular pattern 205 may be displayed to a user on a computingsystem and/or communication device. The user interface 200 may be hostedon a webpage that is accessible by a user and/or stored in memory of acomputing system/communication device. An icon, thumbnail, shortcut, orthe like for the user interface 200 may be displayed in a text messagingapplication, an email application, a video communication application, ane-book, and/or the like. A user may select the icon, thumbnail,shortcut, and/or the like to enable the music generation system/methodto determine states contained in communications and/or to display thefull user interface 200 to a user.

Additionally and/or alternatively, the computing system and/orcommunication device may use the circular pattern 205 to determine alocation of a particular state represented by an icon/position on thecircle without displaying the wheel 205 to the user. An icon, thumbnail,shortcut, or the like for the user interface 200 may be displayed in atext messaging application, an email application, a video communicationapplication, an e-book, and/or the like. A user may select the icon,thumbnail, shortcut, and/or the like to enable the music generationsystem/method to determine states contained in communications.

The wheel 205 is a circular wheel interface where a range of statesand/or other elements/concepts (represented by icons 210) are mapped todifferent positions. The wheel 205 is the source of control for musicparameters and an icon's position on the wheel determines changes in anote pattern, a harmony, a tone, an orchestration, a speed, a rhythm, avolume, and/or the like. The position of each icon on the wheel 205 isthe main control over time of the generation of music. Different regionson the wheel 205 map to noticeable differences in the music that isgenerated by the computing system. A specific intensity and valencescore may determine a state region and/or other element region on thecircular or X/Y grid (shown in FIG. 9D) interface and each state regionand/or other element region may correspond to particular musicalparameters in the generated music.

As shown in FIG. 2B, the plurality of icons 210 and/or positions ofstates may be organized such that similar states and/or otherelements/concepts (represented by icons 210) are grouped together in aparticular region of the circular wheel 205. Similar states may havesimilar musical/sound characteristics to aid in smoothly transitioningbetween states. If a user and/or communication were to traverse aroundan entire circumference of the circular pattern, a beginning point andan end point would generate audio having similar characteristics becausethe beginning point and the end point are located in a similar region.States and/or other elements/concepts grouped together in a particularregion may sound similar but have some noticeable differences. Forexample, a position of state on a circumference of the circular patternor an angle of a position of a state on the circular pattern maycorrespond to a first subset of characteristics associated with aparticular state and/or other element/concept represented by aparticular position, while a distance of position of a state from acenter of the circular pattern may correspond to a second set ofcharacteristics associated with the particular state. Additionallyand/or alternatively, a position of a state on a circumference of thecircular pattern or an angle of a position on the circular pattern maycorrespond to a particular musical arrangement associated with aparticular state and/or other element/concept represented by aparticular position, while a distance of position of a state from acenter of the circular pattern may correspond to an intensity of theparticular musical arrangement.

A user may interact with user interface 200 via tactile input and/ormouse input. Additionally and/or alternatively, the user interface 200may be used by a computing system and/or user device to determine whatmusic to play based on an emotion contained with a communication (e.g.,at least one of a sensor communication, an IoT communication, abiometric/health communication, a voice communication, a textualcommunication, a photographic communication, a video communication,and/or the like). A computing system may determine where a particularposition associated with the determined state is on the wheel 205 andaccess and play the algorithm associated with the particular state basedon where the state is located on the wheel 205.

Based on the user interaction with the user interface 200 and/or basedon a determined state from a communication, the computing system maydetermine the position on a circumference of the wheel 205 of an icon210 and/or position associated with the determined at least oneparticular state or an angle of the icon 210 and/or position associatedwith the determined at least one particular state on the circularpattern and the distance of the particular icon 210 and/or positionassociated with the determined at least one particular state from thecenter of the wheel 205. Based on the determination of the position orangle and the distance of the icon 210 associated with the determined atleast one state the computing system and/or user device may generatemusic having the musical arrangement and the intensity associated withthe determined at least one state of the communication.

Additionally and/or alternatively, the wheel 205 may facilitatetransitioning between music associated with a first particular state tomusic associated with a second particular state. This is shown in FIG.2C. In FIG. 2C, icons 210 and/or positions associated with a particularstate are represented by circles 210 a. A user may first indicate thestate represented by an icon located at the top of wheel 205 via touchinput, mouse input, and/or by indicating a state in a communication.

If the selection is made via touch or a mouse and after a first userindication is made, then a computing system and/or communication devicemay track a user's interaction with the user interface 200 and wheel205. A user may continue to drag his or her finger or other device(e.g., mouse, stylus, and/or the like) along a path 215 on the wheel 205indicating that the music that is generated should transition betweenmusic associated with at least two states. The music that is generatedmay transition in real time or near real time based on the user'sinteraction with the wheel 205. For example, if the user stops for aperiod of time on a particular icon associated with a state, then themusic associated with that particular state may play for that period oftime until the user stops selecting the icon and/or moves on to adifferent state. Additionally and/or alternatively, each icon that theuser selects by dragging his or her finger or other device (e.g., mouse,stylus, and/or the like) along a path 215 may play music associated withthat particular icon/state/position for a predetermined amount of time(e.g., until a note is done playing, until a sequence of notes iscomplete, and/or the like) before transitioning on to the musicassociated with the next selected icon/state/position. In some cases,the predetermined amount of time may be selected by a user of userinterface 200.

The computing system and/or user device may also introduce some lag sothat music that is generated by user interaction is not generated inreal time. For example, if the user drags his or her finger or deviceover user interface 200 quickly, the computing system may not be able totransition between music associated with different states smoothly. Byintroducing lag, the computing system may smoothly transition betweenmusic associated with different emotions.

Additionally and/or alternatively, if a user picks up his finger and/ordevice (220), a computer may determine that a user would like to pausebetween transitioning between a first selected state and an additionalselected state. The music may turn off or continue to play in staticposition. If the user rests on a particular state, the music can stillbe continuously generated.

If a selection of more than one state is made via a communication (e.g.,at least one of a sensor communication, an IoT communication, abiometric/health communication, a voice communication, a textualcommunication, a photographic communication, a video communication,and/or the like) and the communication contains at least two states,then a computing device may determine a path 215 between a firstselected state and at least one additional state. Using the statesbetween the first selected state and at least one additional state, thecomputing system may smoothly transition between playing musicassociated with each of the at least two states contained within thecommunication by playing music associated with the determined additionalstates between music associated with the at least two particular statesindicated in the communication.

Additionally and/or alternatively, the generated music may transitionbetween music associated with the at least two states indicated in thecommunication by pausing music associated with a first indicated statebefore playing music associated with a second indicated state.

The above referenced transition methods may additionally be used toguide a user from a first state to a second desired state. The generatedmusic may continuously guide, based on input from the one or morecommunications, a user from a first state to a second desired state. Ina non-limiting example, a computing device may determine a path 215between a first state and at least one additional desired state. Usingthe states between the first selected state and at least one additionalstate, the computing system may smoothly transition, based on input fromthe one or more communications, between playing music associated witheach of the at least two states contained within the communication byplaying music associated with the determined additional states betweenmusic associated with the at least two particular states indicated inthe communication to guide a user towards the second desired state.

User interface 200, a computing system, and/or a communication devicemay give a user an option to save music that is generated by userinteraction with interface 200 and/or music that is generated byanalyzing the communication. A user may then playback the saved musicthat was previously generated. Additionally and/or alternatively, a usermay be given the option to share (via social media site, text, email,phone, and/or the like) the generated music with others. The user mayfurther be able to create a playlist of generated music and/or give thegenerated music a unique name.

FIG. 3 is an additional schematic diagram illustrating a system 300 forimplementing the generation of music, in accordance with variousembodiments. System 300 may be similar to system 100 of FIG. 1 andperform similar functions as system 100 of FIG. 1. Additionally and/oralternatively, system 300 may be used in conjunction with user interface200 of FIG. 2.

System 300 may comprise computing system 305 (which may correspond tocomputing system 105 of FIG. 1). Computing system 305 may runapplications such as text, voice, and/or video applications and/orreceive input from input devices 310. Input devices 310 may include oneor more sensor devices including one or more IoT sensors, one or morebiometric/health sensor devices 310, text/speech device 310 b, VR/ARdevices, fitness tracker devices, smart watches, EEG devices, one ormore cameras, one or more facial recognition devices, and/or the like.Computing system 305 may also receive direct user input (control) 335via touch, mouse, video game context and/or the like.

Computing system 305, based on the input received from input devices 310and/or direct user input 335, may determine a state contained within theinput received from input devices 310 and/or user input 335. Based onthe determined state, the user interface may determine apositivity/negativity (valence) and/or intensity of music associatedwith the determined state (block 315). The positivity/negativity and/orintensity of the music are discussed further with respect to FIG. 4.Additionally and/or alternatively, computing system 305 may determine aposition of a determined state on the circular pattern 325 or X/Y planeto determine the positivity/negativity and/or intensity of musicassociated with the determined state. Different combinations ofpositivity/negativity and/or intensity characterize different state andmap to various regions around the circular pattern's circumference orposition on an X/Y plane.

In addition to determining a state contained within input devices 310and/or indicated by user input 335, the computing system may furtherdetermine user preferences, demographics, and/or the like whendetermining what music to generate (block 320). A user may directlyenter user preferences, demographics, and/or the like. The computingsystem 305 may also indirectly determine user preferences, demographics,and/or the like. For example, computing system 305 may determine userhabits (i.e. what types of music a user typically listens to and/orbuys).

Additionally and/or alternatively, music generationparameters/characteristics may be adjusted to fit output of theapplication, game, sensor system, and/or the like. For example, if themusic generation system is being used to supplement a voiceconversation, then the music that is generated may contain lowertones/volumes. If the music generation system is being used tosupplement information received from a fitness tracker, then the musicthat is generated may contain higher tones/volumes.

The determined positivity/negativity and/or intensity of the state maybe mapped to a circular pattern 325 (which may correspond to userinterface 200 of FIG. 2) and/or an XY graph (which may correspond to XYgraph 900 d of FIG. 9D). States having similar positivity/negativityand/or intensity music may be located in a similar region of thecircular pattern 325 and/or graph. Additionally, different statecategories may have specific positivity/negativity and/or intensityvalues.

User interface 325 may be displayed to a user and/or invisible to a userdepending on the application of the music generation system 300. Forexample, if computing system 305 is analyzing text and/or voice input,then the interface 325 may be invisible. If the computing system 305 isreceiving user input via touch or mouse, then the interface 325 may bedisplayed to a user.

After determining the positivity/negativity and/or intensity of music,the generated music may be outputted via audio output 330 (which maycorrespond to playback devices 120 of FIG. 1). In some embodiments, theaudio output 330 might each include, without limitation, one or morespeakers external to but communicatively coupled to the computing system305 and/or input device 310, one of one or more speakers integratedwithin the computing system 305 and/or input device 310, one or moreheadphones, one or more earbuds, one or more sound bars, one or morewireless speakers, or one or more stereo speakers, and/or the like.

FIGS. 4A-4C (collectively, FIG. 4) represents a system 400 for mappingmusic to states, in accordance with various embodiments. FIG. 4Aincludes a table 405 for mapping music to different state categories410. FIG. 4A has two mapping parameters positivity 415 a/negativity 415b and/or intensity 420. The negativity scale 415 b is the same aspositivity scale 415 a, except it is reversed (i.e. on the positivityscale “Happy” is given a 10 while on the negativity scale “Happy” isgiven a 0). The positivity/negativity parameters may be collectivelyreferred to as positivity/negativity parameters 415. Although only twomapping parameters are shown (positivity/negativity parameters 415and/or intensity parameter 420) in FIG. 4A, more than two parameters maybe used to map music to different states. Additionally and/oralternatively, only one parameter, such as positivity 415 a, may be usedto map music to states.

Positivity/negativity parameters 415 are rated on a scale of 1-10 whileintensity 420 is rated on a scale of 1-100. These values are rescaledbetween 0-1 then rescaled to control various musical parameters. Statecategories 410 having a similar positivity/negativity parameter 415and/or a similar intensity rating 420 may be grouped together on acircular interface and/or a graph (which may correspond to circularpattern 205 of FIG. 2, circular pattern 325 of FIG. 3, graphs of FIGS.4B and 4C, graph 900 d of FIG. 9D, and/or the like). Thus, the statecategories 410 that are grouped together will generate similar musicwith subtle differences.

Positivity/negativity parameters 415 may correspond to a note pattern, anote probability, an envelope, a harmony, a tone, a filter cut-off,pitch, contour, arpeggiation rate, arpeggiation step size, vocal-likeinflections, elements of surprise, note envelopes, envelope controlsignals, randomization, consistency, an orchestration, a speed, arhythm, a volume, and/or the like associated with a state. Intensityparameters 420 may correspond to a note pattern, a harmony, a tone,crescendo, decrescendo, vocal formant-like filtering or expressiveattribute (e.g., inflection), an orchestration, a speed, a volume,and/or the like associated with a state.

Each value that is given to a particular state may be used to generatemusic that is unique to each particular state. The music that isgenerated is represented by music attributes column 425. For example, ifthe computing system determines that a communication contains the state“Happy” and/or a user has selected an icon associated with the state“Happy,” the computing system may generate music having a major mode,fast tempo, staccato, quick attack (note onsets), short decay, andbright timbre.

Music attributes column 425 may further vary based on the type ofapplication/communication (e.g., voice, text, speech, fitness tracker,EEG device, smart watch, AR/VR device, user input, and/or the like), thedemographics (e.g., age, sex, and/or the like) of the user, the user'spreferences, and/or the like.

FIGS. 4B and 4C further represent ways to map different music parametersto states using graphs. FIG. 4B represents mapping positivity/negativityparameters 415 to different musical characteristics. In a non-limitingexample, states mapped to positive parameters 415 a (e.g., “Happy”) maycause the computing system to generate music that has a brighter timbre,a major mode, higher note density, more implied polyphony, steadyrhythm, smooth curves, less randomness, larger jumps, and/or the like.While states mapped to negative parameters 415 b (e.g. “Sad”), may causethe computing system to generate music that has a darker timber, a minormode, more chromaticism, higher dissonance, more irregular curves,jagged pitch curves and/or the like.

FIG. 4C represents mapping intensity parameters 420 to different musicalcharacteristics. In a non-limiting example, states mapped to highintensity parameters 420 a (e.g. “Excited,” “Anxious,” and/or the like)may cause the computing system to generate music that has largeamplitude modulation, fast arpeggiation rate, high brightness, highpitch height, and/or the like. While states mapped to low intensityparameters 420 b (e.g., “Calm,” “Dreamy,” and/or the like), may causethe computing system to generate music that has low amplitudemodulation, slow arpeggiation rate, long note envelope, low pitch range,and/or the like.

Users/communications may interact with graphs of FIGS. 4B and 4C in asimilar manner as user interface 200 of FIG. 2 to generate music.

Each of the examples described above, with respect to FIG. 4, isintended to be non-limiting. A variety of parameters (instead ofpositivity/negativity and/or intensity) may be used to create musicassociated with each state. Further, positivity/negativitycharacteristics and/or intensity characteristics are not limited tothose described above. Each state could have an almost unlimited numberof musical characteristics associated with it and the musicalcharacteristics associated with each emotion are not limited to thosementioned above.

FIG. 5 is a flow diagram illustrating a method 500 for implementing thegeneration of music, in accordance with various embodiments.

While the techniques and procedures are depicted and/or described in acertain order for purposes of illustration, it should be appreciatedthat certain procedures may be reordered and/or omitted within the scopeof various embodiments. Moreover, while the method 500 illustrated byFIG. 5 can be implemented by or with (and, in some cases, are describedbelow with respect to) the system 100 of FIG. 1 (or components thereof),the user interface 200 of FIG. 2 (or components thereof), the system 300of FIG. 3 (or components thereof), the mapping system 400 of FIG. 4 (orcomponents thereof), and/or the mapping system of FIG. 9 (or componentsthereof), such methods may also be implemented using any suitablehardware (or software) implementation. Similarly, while each of thesystem 100 of FIG. 1 (or components thereof) user interface 200 of FIG.2 (or components thereof), the system 300 of FIG. 3 (or componentsthereof), the mapping system 400 of FIG. 4 (or components thereof),and/or the mapping system of FIG. 9 (or components thereof), can operateaccording to the method 500 illustrated by FIG. 5 (e.g., by executinginstructions embodied on a computer readable medium), the system 100 ofFIG. 1 user interface 200 of FIG. 2, the system 300 of FIG. 3, themapping system 400 of FIG. 4, and/or the mapping system of FIG. 9 caneach also operate according to other modes of operation and/or performother suitable procedures.

Although method 500 is described with respect to emotions/concepts, asimilar method may be used for different states of an environment ordifferent states of a user which may include at least one of an emotionof a user, a feeling of a user, a location of the user a, a physicalposition of a user, a level of activity of a user, an action of a user,and/or the like. In the non-limiting embodiment of FIG. 5, method 500,at block 505, might comprise analyzing, with a computing system, acommunication to determine at least one emotion (concept or state)contained within the communication. The computing system may be at leastone of a desktop computer, a laptop computer, a tablet, a smart phone,an e-reader, and/or the like. Additionally and/or alternatively, in someembodiments, the computer system might be embedded in an exercisemachine, physical therapy device, headphone, wristband or headband.Additionally and/or alternatively, in some embodiments, the computingsystem might include, without limitation, one of a processor of aset-top box, a processor of a digital video recording (“DVR”) device, aprocessor of a user device running a software application (“app”), aprocessor of an audio playback device, a processor on an input device(e.g., fitness tracker, EEG device, or the like) running an app, aprocessor of a media player, a processor of a gaming console, aprocessor in audio equipment, and/or the like. The concept may be atleast one of an emotion of a person, a state of a person, an action of aperson, or a scene.

The communication may be at least one of a sensor communication, an IoTcommunication, a biometric/health communication, a voice communication,a textual communication, a photographic communication, a videocommunication, a VR/AR communication, and/or the like.

The sensor communication may contain feedback from one or more sensorsincluding, but not limited to, one or more distance sensors, one or moremotion sensors, one or more movement sensors, one or more speed orvelocity sensors, one or more accelerometer sensors, one or morebiometric/health sensors, one or more facial recognition sensors, one ormore camera sensors, one or more IoT sensors (e.g., thermometers,humidity sensors, etc.) and/or the like. Based on input from the one ormore sensors, the computing system 105, input device 115, and/or atleast one audio playback device 120 may determine a mental or physicalstate that a person is experiencing or a state of the environment.

The IoT communication may contain feedback from one or more IoT sensorscontained within a home. For example, the one or more IoT sensors mightinclude one of one or more thermometers in one or more rooms, one ormore infrared (“IR”) thermometers aimed at one or more positions in theone or more rooms, one or more air flow sensors in the one or morerooms, one or more air flow sensors in air ducts directed toward the oneor more rooms, one or more indoor solar light sensors, one or moreoutdoor solar light sensors, one or more outdoor wind sensors, one ormore neighborhood weather station sensors, one or more regional weatherstation sensors, one or more motion detectors detecting presence ofpeople or animals in at least one of the one or more rooms or outsidethe customer premises, one or more humidity sensors in the one or morerooms, one or more smoke detectors detecting smoke in the one or morerooms, one or more gas detection sensors detecting gas in the one ormore rooms, one or more biometric sensors identifying at least oneperson, or one or more health sensors detecting health information forat least one person, and/or the like. Based on input from the one ormore sensors, the computing system, input device, and/or at least oneaudio playback device may determine a state that a person isexperiencing or a state of the environment.

The biometric/health communication may be received from at least one ofa fitness tracker or an electroencephalography (“EEG”) device and thecomputing system may determine the at least one emotion of a personand/or concept based on feedback from the at least one of the fitnesstracker or the EEG device. In a non-limiting example, the fitnesstracker and/or EEG device may provide feedback about a blood pressure ofa user, a heart rate of a user, electrical brain waves of a user, and/orthe like. Based on the blood pressure, the heart rate, the electricalbrain waves and/or the like, the computing system may determine anemotion/concept that a user is experiencing. Merely by way of example,if the user has an elevated blood pressure and a high heart rate thecomputing system may determine that a user of the fitness tracker and/orEEG device is feeling stressed.

The voice communication may be received via user input or from a phonecall between a calling party and a called party. In order to determinethe at least one emotion/concept of a person, the computing system mayparse the voice communication of the at least one person and/ordetermine a tone of voice of at least one person. Merely by way ofexample, if the communication is via phone, the computing system mayparse the voice communication between the calling party and the calledparty to determine how each party is feeling. Additionally and/oralternatively, the computing system may analyze the tone of voice ofeach party to determine what emotions/state each party is experiencing.

With regard to the textual communications, the computing system mayanalyze text messages, instant messages, social media posts, emails,books, and/or the like to determine an emotion/concept that auser/person is experiencing. The computing system may parse the text ofthe textual communication to determine the emotion/concept of a person.Merely by way of example, the computing system may parse the text forkey words such as “happy” or “sad” and/or the computing system may parsethe textual communication for emojis to determine the mood of a person.Additionally, the computing system may parse the words of a book todetermine a scene (e.g., action scene, adventure scene, romance scene,etc.) of a book or an action/state of a person in the book.

The photographic communication may be a photograph taken with a camera.The computing system may then use facial recognition tools to determinea displayed emotion/concept of at least one person in the photograph.The video communication may be taken with a video camera. The videocommunication may also be a video phone call between at least twoparties. The computing system may then use facial recognition tools todetermine a displayed emotion/concept of the at least one person in thevideo. The computing system may also use tools to analyze body languageof the at least one person in the video communication to determine anemotion/concept that the at least one person is experiencing. Thecomputing system may further parse the dialogue of the at least oneperson in the video or analyze the tone of voice of at least one personin the video communication to determine an emotion/concept that the atleast one person in the video is experiencing. Additionally and/oralternatively, the computing system may use facial recognition tools,analyze body language, parse dialogue, analyze tone of voice todetermine a state/action (e.g., running, walking, and/or the like) of aperson and/or a scene (e.g., action scene, adventure scene, romancescene, and/or the like) of the video/picture.

A VR communication and/or AR communication may be obtained from VR/ARdevices (e.g., cell phones, tablets, headsets, glasses, goggles, lenses,and/or the like). The computing system 105, input device 115, and/or atleast one audio playback device 120 may analyze the facial expression,body language, and/or the like of the user of the AR/VR device and/orone or more persons interacting/communicating with the user of the AR/VRdevice. The computing system 105, input device 115, and/or at least oneaudio playback device 120 may also parse the dialogue or determine atone of voice of the user and/or one or more persons interacting withthe user of the AR/VR device. The facial expression, body language,dialogue, and/or tone of voice may then be used to determine aconcept/emotion of a user and/or person. A state of a virtual conceptdefined within the game metrics may also be used to drive the state ofthe music system.

The method 500, at block 510, may further comprise autonomouslydetermining, with the computing system, one or more firstcharacteristics of a plurality of characteristics of music associatedwith the determined at least one emotion/concept/state contained withinthe communication. The one or more first characteristics of theplurality of characteristics of music include at least one of a notepattern, a harmony, a tone, an orchestration, a speed, a rhythm, avolume, and/or the like associated with the first emotion/conceptindicated by the communication.

At block 515, method 500 may further comprise, based on thedetermination of the one or more first characteristics of a plurality ofcharacteristics of music associated with the at least oneemotion/concept/state contained within the communication, autonomouslygenerating, with the computing system, music having the one or morefirst characteristics of the plurality of characteristics associatedwith the at least one determined emotion/concept contained within thecommunication.

Additionally and/or alternatively, the communication may furtherindicate at least one of an age and/or sex of a person. The one or morefirst characteristics of a plurality of characteristics of music mayfurther be associated with the at least one of the age or the sexindicated by the communication. The music that is generated further hasthe one or more first characteristics of the plurality ofcharacteristics associated with the at least one of the age or the sexindicated by the communication.

The generated music may further have human-like embellishments. Thehuman-like embellishments may be created from at least one of timingjitter, frequency jitter, timbral jitter, and/or the like. These allowfor the music to continuously evolve even when stuck in one state.

Human performers have a natural imprecision which must be explicitlyaccounted for in computer generated music. To do this, irregularmicro-fluctuations may be added to the timing of note onsets. This kindof small random signal bias is often referred to as “timing jitter.” Asa result, quantized notes are gently un-quantized to provide a morepleasing and human-sounding musical aesthetic. Timing jitter providessubtle rhythmic variation that may add nuance to the static notepatterns.

Similar to jittered timing offsets (“timing jitter”), “frequency jitter”may be utilized to modulate the frequency (pitch) of the generatedmusic. Depending on the duration and articulation of the note, frequencyjitter parameters will change. For instance, long sustained notes willbe subject to more evolved jitter (gradual drift), a technique to addwarmth; while shorter, more percussive notes will have little to nojitter.

Jitter may also be mapped to a number of other parameters in charge ofproducing timbre or the sound qualities of the notes. This is referredto as “timbral jitter.” These parameters exist due in part to thereal-time audio synthesis engine, which allow dynamic control of a soundover time via digital signal processing.

Additional embodiments of method 500 may further comprise an initialstep of mapping, with the computing system, a plurality of icons to acircular pattern, wherein each icon of the plurality of icons isassociated with at least one emotion/concept/state, wherein at least oneof a position of an icon on a circumference of the circular pattern oran angle of the icon on the circular pattern corresponds to a firstsubset of particular characteristics of music associated with aparticular emotion/concept/state represented by the icon, and wherein adistance of the icon from a center of the circular pattern correspondsto a second subset of particular characteristics of music associatedwith the particular emotion/concept/state represented by the icon (block520). This map of the plurality of icons may be displayed to the user orthe circular pattern may merely be used by the computer to mapparticular emotions/concepts to a particular musical generationalgorithm.

After determining at least one emotion/concept/state contained withinthe communication (block 505), the method may further determine theposition on the circumference of the circular pattern of at least oneicon associated with the determined at least one particularemotion/concept/state or the angle of the at least one icon associatedwith the determined at least one particular emotion/concept/state on thecircular pattern and the distance of the particular icon associated withthe determined at least one particular emotion/concept/state/adjectivefrom the center of the circular pattern (block 525). Based on thedetermination of the position or angle and the distance of the at leastone icon associated with the determined at least oneemotion/concept/state, the method, at block 550, may autonomouslygenerate music having the first subset of particular characteristics andthe second subset of particular characteristics associated with thedetermined at least one icon associated with the at least one determinedemotion/concept/state of the communication.

FIG. 6 is a flow diagram illustrating a method 600 for implementing auser interface for the generation of music, in accordance with variousembodiments.

While the techniques and procedures are depicted and/or described in acertain order for purposes of illustration, it should be appreciatedthat certain procedures may be reordered and/or omitted within the scopeof various embodiments. Moreover, while the method 600 illustrated byFIG. 6 can be implemented by or with (and, in some cases, are describedbelow with respect to) the system 100 of FIG. 1 (or components thereof),the user interface 200 of FIG. 2 (or components thereof), the system 300of FIG. 3 (or components thereof), the mapping system 400 of FIG. 4 (orcomponents thereof), and/or the mapping system of FIG. 9 (or componentsthereof), such methods may also be implemented using any suitablehardware (or software) implementation. Similarly, while each of thesystem 100 of FIG. 1 (or components thereof) user interface 200 of FIG.2 (or components thereof), the system 300 of FIG. 3 (or componentsthereof), the mapping system 400 of FIG. 4 (or components thereof),and/or the mapping system of FIG. 9 (or components thereof), can operateaccording to the method 600 illustrated by FIG. 6 (e.g., by executinginstructions embodied on a computer readable medium), the system 100 ofFIG. 1 user interface 200 of FIG. 2, the system 300 of FIG. 3, themapping system 400 of FIG. 4, and/or the mapping system of FIG. 9 caneach also operate according to other modes of operation and/or performother suitable procedures.

Although method 600 is described with respect to emotions/concepts, asimilar method may be used for different states of an environment ordifferent states of a user which may include at least one of an emotionof a user, a mental state of a user, a feeling of a user, a physicalstate of a user, a location of the user a, a physical position of auser, a level of activity of a user, an action of a user, or differentstates of an environment, and/or the like. In the non-limitingembodiment of FIG. 6, method 600, at block 605, might comprisegenerating with a computing system, a circular pattern having aplurality of different emotions represented by icons, wherein at leastone of a position of a particular icon on the circular patterncorresponds to a set of characteristics of a plurality ofcharacteristics of music associated with a particular emotionrepresented by the particular icon. The set of characteristics maycorrespond to positivity/negativity and/or intensity described withrespect to FIGS. 3 and 4 and/or any other characteristics of music. Atblock 610, method 600 may further analyze, with a computing system, acommunication to determine at least one emotion contained within thecommunication.

Additionally and/or alternative, the method 600, at block 615 mayautonomously determine, with the computing system, the position of atleast one icon corresponding to the determined at least one emotioncontained within the communication on the circular pattern. The method600 may then, based on the determination of the position of thedetermined at least one particular icon on the circular pattern,autonomously generate, with the computing system, music having the setcharacteristics associated with the determined at least one emotioncontained within the communication (block 620).

FIGS. 7A-7C (collectively, FIG. 7) are flow diagrams illustrating amethod 700 for implementing a user interface for the generation ofmusic, in accordance with various embodiments.

While the techniques and procedures are depicted and/or described in acertain order for purposes of illustration, it should be appreciatedthat certain procedures may be reordered and/or omitted within the scopeof various embodiments. Moreover, while the method 700 illustrated byFIG. 7 can be implemented by or with (and, in some cases, are describedbelow with respect to) the system 100 of FIG. 1 (or components thereof)the user interface 200 of FIG. 2 (or components thereof), the system 300of FIG. 3 (or components thereof), the mapping system 400 of FIG. 4 (orcomponents thereof), and/or the mapping system of FIG. 9 (or componentsthereof), such methods may also be implemented using any suitablehardware (or software) implementation. Similarly, while each of thesystem 100 of FIG. 1 (or components thereof) user interface 200 of FIG.2 (or components thereof), the system 300 of FIG. 3 (or componentsthereof), the mapping system 400 of FIG. 4 (or components thereof),and/or the mapping system of FIG. 9 (or components thereof), can operateaccording to the method 700 illustrated by FIG. 7 (e.g., by executinginstructions embodied on a computer readable medium), the system 100 ofFIG. 1, user interface 200 of FIG. 2, the system 300 of FIG. 3, themapping system 400, and/or the mapping system of FIG. 9 can each alsooperate according to other modes of operation and/or perform othersuitable procedures.

Although method 700 is described with respect to emotions/concepts, asimilar method may be used for different environmental states ordifferent states of a user which may include at least one of an emotionof a user, a feeling of a user, a location of the user a, a physicalposition of a user, a level of activity of a user, an action of a user,and/or the like. In the non-limiting embodiment of FIG. 7, method 700,at block 705, may comprise generating, with a computing system, acircular pattern having a plurality of different emotions represented byicons (or a position on the circular pattern), wherein at least one of aposition on a circumference of the circular pattern or an angle of thecircular pattern corresponds to a first subset of characteristics of aplurality of characteristics of music associated with a particularemotion represented by a particular icon (or position), and wherein adistance from a center of the circular pattern corresponds to a secondsubset of characteristics of the plurality of characteristics of musicassociated with a particular emotion represented by a particular icon.

Although icons are used in the method described below, states oremotions may also be mapped to particular regions/positions and theregions/positions may be used instead of icons to generate music.

Instead of a circular pattern, a graph may also be used. The method 700might alternatively include generating, with a computing system, a graphhaving a plurality of different states mapped to different positions onthe graph. An x-axis might correspond to a first subset ofcharacteristics of a plurality of characteristics of music while ay-axis might correspond to a second subset of characteristics of theplurality of characteristics of music. In this way each axis serves as aseparate input into the generative music system. Moving along the X-axisand/or Y axis might cause the first or second parameters of music tochange. Based on a determination of the position of a first state, thecomputing system may synthesize the one or more first subset ofcharacteristics of music together with the one or more secondcharacteristics of music.

In a non-limiting example, the angle of an icon/position of a stateand/or the position of an icon on the circumference of the circularpattern may correspond to a positivity/negativity parameter of music(described with respect to FIGS. 3 and 4) while the distance of the iconfrom the center of the circle may correspond to an intensity parameterof music (described with respect to FIGS. 3 and 4). Additionally and/oralternatively, the angle of an icon and/or the position of an icon onthe circumference of the circular pattern may correspond to an intensityparameter of music (described with respect to FIGS. 3, 4, and 9) whilethe distance of the icon from the center of the circle may correspond toa positivity/negativity parameter of music (described with respect toFIGS. 3, 4, and 9).

The music associated with the emotions is not limited to onlypositivity/negativity and/or intensity parameters. A variety ofparameters (instead of positivity/negativity and/or intensity) may beused to create music associated with each emotion. There is virtually anunlimited number of ways to position icons associated with emotions onthe circular pattern and group icons associated with emotions togetherin a particular region on the circular pattern.

The computing system may be at least one of a desktop computer, a laptopcomputer, a tablet, a smart phone, an e-reader, and/or the like.Additionally and/or alternatively, in some embodiments, the computingsystem might include, without limitation, one of a processor of aset-top box, a processor of a digital video recording (“DVR”) device, aprocessor of a user device running a software application (“app”), aprocessor of an audio playback device, a processor on an input device(e.g., fitness tracker, EEG device, or the like) running an app, aprocessor of a media player, a processor of a gaming console, aprocessor in audio equipment, and/or the like.

The icons used to represent the different emotions may be at least oneof a text icon that represents the particular emotion (e.g., “HAPPY,”“SAD,” or the like) and/or an emoji (e.g., C)) that represents aparticular emotion. Additionally and/or alternatively, states may bemapped to particular regions of a circular pattern or graph without theuse of icons and music may be generated based on the position of thestate within the circular pattern or graph.

The first subset of characteristics of music may include at least one ofa note pattern, a harmony, a tone, an orchestration, a speed, a volume,and/or the like associated with the first emotion indicated by the usercommunication. The second subset of characteristics may include at leastone of a note pattern, a harmony, a tone, an orchestration, a speed, avolume, and/or the like associated with the first emotion indicated bythe user communication. In a non-limiting example, the position of anicon on the circumference of the circle and/or an angle of the icon maycorrespond to a particular musical/note arrangement and/or musicalalgorithm and all emotions that are located in a position/angle may havethe same or similar musical/note arrangement and/or musical algorithm. Adistance from center may correspond to a particular volume or speed ofthe same or similar musical/note arrangement and/or musical algorithm.Thus, each icon on the circular pattern will be different (if onlyslightly) from every other icon on the circular pattern.

Method 700, at block 710, may further comprise analyzing, with acomputing system, a communication to determine at least one emotioncontained within the communication. The communication may be at leastone of an IoT communication, biometric/health communication, a voicecommunication, a textual communication, a photographic communication, avideo communication, a tactile communication, and/or the like.

The biometric/health communication may be received from at least one ofa fitness tracker or an electroencephalography (“EEG”) device and thecomputing system may determine the at least one mental state of a personbased on feedback from the at least one of the fitness tracker or theEEG device. In a non-limiting example, the fitness tracker and/or EEGdevice may provide feedback about the attention level, relaxation level,a blood pressure of a user, a heart rate of a user, electrical brainwaves of a user, and/or the like. Based on the mental state, bloodpressure, the heart rate, the electrical brain waves and/or the like,the computing system may determine a mental state that a user isexperiencing. Merely by way of example, if the user has an elevatedblood pressure and a high heart rate the computing system may determinethat a user of the fitness tracker and/or EEG device is feelingstressed.

The voice communication may be received via user input or from a phonecall between a calling party and a called party. In order to determinethe at least one emotion of a person, the computing system may parse thevoice communication of the at least one person and/or determine a toneof voice of at least one person. Merely by way of example, if thecommunication is via phone, the computing system may parse the voicecommunication between the calling party and the called party todetermine how each party is feeling. Additionally and/or alternatively,the computing system may analyze the tone of voice of each party todetermine what emotions each party is experiencing.

With regard to the textual communications, the computing system mayanalyze text messages, instant messages, social media posts, emails,books, and/or the like to determine an emotion that a user isexperiencing. The computing system may parse the text of the textualcommunication to determine the emotion of a person. Merely by way ofexample, the computing system may parse the text for key words such as“happy” or “sad” and/or the computing system may parse the textualcommunication for emojis to determine the mood of a person.

The photographic communication may be a photograph taken with a camera.The computing system may then use facial recognition tools to determinea displayed emotion of at least one person in the photograph. The videocommunication may be taken with a video camera. The video communicationmay also be a video phone call between at least two parties. Thecomputing system may then use facial recognition tools to determine adisplayed emotion of the at least one person in the video. The computingsystem may also use tools to analyze body language or gestures of the atleast one person in the video communication to determine an emotion thatthe at least one person is experiencing. The computing system mayfurther parse the dialogue of the at least one person in the video oranalyze the tone of voice of at least one person in the videocommunication to determine an emotion that the at least one person inthe video is experiencing.

The tactile communication may be received via user input from a touchscreen and/or a mouse. The user may select one or more iconsrepresenting an emotion on a display that is displaying the circularpattern of emotions to generate music associated with that emotion. Thecomputing system may determine whether the user selected at least twoicons by tracking the tactile communication of the user with the displaydevice. Based on a determination that at least two icons have beenselected, the computing system may determine whether the tactile input,when selecting the at least two icons, was continuous input. In otherwords, continuous input would occur when a user remains in constanttactile contact with the display device and/or if the user holds downthe left button of the mouse. Non-continuous input would occur if theuser lifts his or her finger/hand from the display device and/or lets goof the left mouse button. Based on a determination that the tactileinput was not continuous, the method 700 may pause the generated musicbetween each user selection of an icon. Based on a determination thatthe tactile input was continuous, the method 700 may transition betweenplaying music associated with each of the at least two emotions selectedby the tactile input. Touching the interface may turn on the sound,while lifting the finger turns it off. The position of the finger,velocity, direction finger is moving may be used to control the soundthat is generated. In other cases, the sound is always on and the fingerinput can be used to influence the sound.

If the computing system cannot keep up with the tactile communication ofthe user (via tactile, mouse, or the like), then the computing systemmay create a time lag between the selection of each icon and thegeneration of music associated with each icon. Additionally and/oralternatively, the computing system may play the generated music foreach selected icon for a predetermined amount of time beforetransitioning to music associated with a subsequent selected icon.

Interpolation, both smooth and quantized, may also be used to allow forthe gradual transitioning between music associated with at least twoemotions. In other words, once the computing system determines that anew emotion has been included in a communication and/or selected by auser, different aspects of the music will the transition between thestates associated between the at least two emotions. Unusually jumpyrapid, jumpy movements in the communication, signal noise, and/orindicated by user selection (unexpectedly rapid signals) will be handledvia standard smoothing filters for control signals.

Depending on the type of communication the computing system receives,the circular pattern may be displayed to the user on a user interfaceand/or the circular pattern may merely be used by the computing systemto create differences between music associated with a particularemotion. For example, if the computing system is receiving a tactilecommunication, then the tactile input may be received on a userinterface that is displaying the communication. However, if thecomputing system is receiving a voice communication, then the computingsystem may determine where the emotion is on the circular pattern todetermine what type of music to generate, without displaying thecircular pattern to the user.

Merely by way of example, method 700, at block 715 may further compriseautonomously determining, with the computing system, the position of atleast one particular icon corresponding to the at least one particularemotion on the circumference of the circular pattern and the distance ofthe at least one particular icon corresponding to the at least oneparticular emotion from the center of the circular pattern. Based on thedetermination of the position of the determined at least one particularicon on the circumference of the circular pattern or the angle of thedetermined at least one particular icon and the distance of thedetermined at least one particular icon from the center of the circularpattern, the computing system may autonomously generate music having thefirst subset of characteristics and the second subset of characteristicsassociated with the at least one emotion contained within thecommunication (block 720).

Additionally and/or alternatively, the communication may furtherindicate at least one of an age and/or sex of a person. The one or morefirst characteristics of a plurality of characteristics of music mayfurther be associated with the at least one of the age or the sexindicated by the communication. The music that is generated further hasthe one or more first characteristics of the plurality ofcharacteristics associated with the at least one of the age or the sexindicated by the communication.

The generated music may further have human-like embellishments. Thehuman-like embellishments may be created from at least one of timingjitter, frequency jitter, timbral jitter, and/or the like.

Human performers have a natural imprecision which must be explicitlyaccounted for in computer generated music. To do this, irregularmicro-fluctuations are added to the timing of note onsets. This kind ofsmall random signal bias is often referred to as “timing jitter.” As aresult, quantized notes are gently un-quantized to provide a morepleasing and human-sounding musical aesthetic. Timing jitter providessubtle rhythmic variation that add nuance to the static note patterns.

Similar to jittered timing offsets (“timing jitter”), “frequency jitter”is utilized to modulate the frequency (pitch) of the generated music.Depending on the duration and articulation of the note, frequency jitterparameters will change. For instance, long sustained notes will besubject to more evolved jitter (gradual drift), a technique to addwarmth to the timbre; while shorter, more percussive notes will havelittle to no jitter.

Jitter may also be mapped to a number of other parameters in charge ofproducing timbre or the sound qualities of the notes. This is referredto as “timbral jitter.” These parameters exist due in part to thereal-time audio synthesis engine, which allow dynamic control of a soundover time via digital signal processing.

Additionally and/or alternatively, a non-limiting embodiment of method700, at block 725, may comprise autonomously determining, with thecomputing system, whether at least two emotions are contained within thecommunication. Based on a determination that at least two emotions arecontained within the communication, method 700 may simultaneouslygenerate, play, and harmonize music associated with each emotion (block730) and/or transition between music associated with each emotion(blocks 735-760).

Merely by way of example, method 700 at block 730 may comprise, based ona determination that at least two emotions are contained within thecommunication, simultaneously generating, playing, and shaping musichaving the particular first subset of characteristics and the particularsecond set of characteristics associated with each of the at least twoemotions contained within the communication. In other words, if morethan one emotion is present in the communication, the music associatedwith each determined emotion may be generated and played at the sametime. The music associated with each emotion may further be harmonizedto ensure that the sound that is generated is pleasing to hear.

Additionally and/or alternatively, method 700, at block 735, may furthercomprise, based on a determination that more than one emotion iscontained within the communication, determining an order to generate andplay music associated with each of the at least two emotions. The ordermay be determined based on the order that the emotions appear in a text,video, and/or voice communication or the order may be based on who isspeaking in a voice or video communication. At block 740, the method 700may transition, with the computing system, between generating andplaying music associated with each of the at least two emotions. Inother words, the method 700 may play music associated with each emotionseparately. Music associated with a particular emotion may be played fora predetermined period of time before transitioning to music associatedwith the next determined emotion.

Additionally and/or alternatively, method 700 may smoothly transitionbetween music associated with each emotion. In order to do this, afterdetermining an order to generate and play music (block 735), method 700,at block 745, may further comprise autonomously determining, with thecomputing system, the position and/or angle of the at least twoparticular icons corresponding to the at least two particular emotionson the circumference of the circular pattern and the distance of theparticular icon corresponding to the particular emotion from the centerof the circular pattern. Method 700 may further autonomously determine,with the computing system, icons corresponding to additional emotionsbetween the at least two particular icons corresponding to the at leasttwo particular emotions (block 750). Method 700, at block 755, may thentransition, with the computing system, between playing music associatedwith each of the at least two emotions contained within thecommunication by playing music associated with the determined additionalemotions between music associated with the at least two particular iconscorresponding to the at least two particular emotions. By playing musicassociated with the determined additional emotions between musicassociated with the at least two particular icons corresponding to theat least two particular emotions, the transition between music may besmoother and less dissonant.

Additionally and/or alternatively, method 700 may synchronize musicassociated with a subset of emotions while transitioning between musicassociated with another subset of emotions.

Method 700 may additionally comprise steps to ensure that the userinterface and music generation system can effectively transition betweenmusic associated each determined emotion. In some situations, a user mayselect icons associated with particular emotions or send communicationcontaining text or emojis representing emotions quicker than thecomputer can generate and output the music, in order to address thosesituations, method 700, at block 760 may comprise receiving, with thecomputing system, at least one additional user communication from auser, the at least one additional user communication being indicative ofat least one additional emotion. Next, the computing system maydetermine a number of user communications over a specified period oftime (block 765) and determine whether the number of user communicationsexceeds a predetermined threshold (block 770).

Based on a determination that the number of user communications does notexceed the predetermined threshold, method 700, at block 775, mayfurther comprise autonomously determining, with the computing system,one or more first characteristics of a plurality of characteristics ofmusic associated with the second emotion indicated by the second usercommunication, and autonomously transitioning from generating musichaving the one or more first characteristics of the plurality ofcharacteristics associated with the first emotion indicated by the userto generating music having the one or more first characteristics of theplurality of characteristics associated with the second emotionindicated by the user. Based on a determination that the number of usercommunications does exceed the predetermined threshold, method 700, atblock 780, may comprise pausing, with the computing system, the musicbeing generated.

FIG. 8 is a flow diagram illustrating a method 800 for generating musicassociated with an emotion, in accordance with various embodiments.

While the techniques and procedures are depicted and/or described in acertain order for purposes of illustration, it should be appreciatedthat certain procedures may be reordered and/or omitted within the scopeof various embodiments. Moreover, while the method 800 illustrated byFIG. 8 can be implemented by or with (and, in some cases, are describedbelow with respect to) the system 100 of FIG. 1 (or components thereof)the user interface 200 of FIG. 2 (or components thereof), the system 300of FIG. 3 (or components thereof), the mapping system 400 of FIG. 4 (orcomponents thereof), and/or the mapping system of FIG. 9 (or componentsthereof), such methods may also be implemented using any suitablehardware (or software) implementation. Similarly, while each of thesystem 100 of FIG. 1 (or components thereof) user interface 200 of FIG.2 (or components thereof), the system 300 of FIG. 3 (or componentsthereof), the mapping system 400 of FIG. 4 (or components thereof),and/or the mapping system of FIG. 9 (or components thereof), can operateaccording to the method 800 illustrated by FIG. 8 (e.g., by executinginstructions embodied on a computer readable medium), the system 100 ofFIG. 1 user interface 200 of FIG. 2, the system 300 of FIG. 3, themapping system 400 of FIG. 4, and/or the mapping system of FIG. 9 caneach also operate according to other modes of operation and/or performother suitable procedures.

Although method 800 is described with respect to emotions/concepts, asimilar method may be used for different states of a user which mayinclude at least one of an emotion of a user, a feeling of a user, alocation of the user a, a physical position of a user, a level ofactivity of a user, an action of a user, and/or the like. In anon-limiting embodiment, method 800, at block 805 may comprise defining,with a computing system, a predetermined set of notes. Next, method 800,at block 810 may comprise defining, with the computing system, at leastone of a note pattern, a note range, a harmony, a tone, anorchestration, a speed, or a volume associated with a particularemotion. At block 815, method 800 may comprise generating, with thecomputing system, music associated with the at least one of the notepattern, the note range, the harmony, the frequencies, theorchestration, the speed, note amplitude envelope or the outputamplitude associated with the particular emotion. At block 820, method800, may comprise adding, with the computing system, human-likeembellishments to the generated music.

FIGS. 9 and 10 represent methods, systems, and apparatuses forexercising continuous control over music generation to guide a usertoward a desired state or goal. Each of the methods, systems, andapparatuses described with respect to FIGS. 9 and 10 may be incorporatedinto the different embodiments described with respect to FIGS. 1-8.Additionally, each of the different embodiments described with respectto FIGS. 1-8 may be incorporated into the different embodimentsdescribed with respect to FIGS. 9 and 10. In a non-limiting example,user interface 200 may be used to implement different embodimentsdescribed in FIGS. 9 and 10. Additionally, graphical interface 900 d maybe used in place of user interface 200 and function in a similar manneras user interface 200. Other patterns/shapes may be used as an interface(e.g., a three-dimensional graph, a triangle, a square, an oval, etc.).

FIGS. 9A-9E are schematic diagrams illustrating systems 900 forcontinuously controlling the generation of music/audio/sound to guide auser toward a desired state, in accordance with various embodiments.FIGS. 9A and 9B are directed towards guiding a user from a firstlocation or first position to a second location or position. FIG. 9Arepresents a system 900 a using one parameter (distance to goal) 905 tocontinuously control the one or more parameters of music/audio/sound.FIG. 9B represents a system 900 b using two parameters (distance to goaland movement direction) 905 and 910 to continuously control one or moreparameters of music/audio/sound. FIG. 9C represents a system 900 c usingthree parameters to continuously control the generation ofmusic/audio/sound. FIG. 9D represents a XY coordinate system 900 d formapping different states to music/audio/sound and for transitioningbetween music/audio/sound associated with different states.

FIGS. 9A and 9B include systems 900 a and 900 b for mappingmusic/audio/sound to different parameters to guide a user toward adesired goal. The different parameters may correspond to a state of auser and may include, but are not limited to, at least one of a distanceto goal, a direction of movement, an amount of movement, a breathingpattern, a heart rate, a state in a video game, a step rate, an exerciserate or pattern, an amount of perspiration, a brainwave pattern, and/orthe like. FIG. 9A has one mapping parameter 905—distance to goal. FIG.9B has two mapping parameters 905 and 910 which (for this model)represent distance to goal and movement direction with respect to goaland previous point, respectively. Although only two mapping parametersare shown in FIG. 9B, more than two parameters may be used to map musicto different states and control the generation of music as shown in FIG.9C. Additionally, one mapping parameter may control one or moremusical/audio/sound elements.

Mapping parameters 905 and 910 may correspond or map to one or moremusical/audio/sound characteristics 915 a-915 n. The different musicalcharacteristics might correspond to a note pattern, a harmony, a tone,pitch, a density, vocal-like qualities, elements of surprise,randomization, consistency, an orchestration, a speed, a rhythm, avolume, note envelope, control signal envelope, rate, amplitude on partof signal, and/or the like designed to guide a user toward a desiredstate.

Each value that is given to a particular state may be used to generateand continuously control music to guide a user toward a desired state(e.g., from a first location to a second location). The music may becontinuously controlled based on one or more sensor inputs. For example,one or more sensors may be used to detect user movement/motion. As auser moves, the sensors may detect whether the user is moving closer toa desired second location or away from a desired second location. Basedon the input from the sensors, the computing system may then generatemusic indicating that the user is moving closer to or away from thedesired second location. The computing system may continuously controland/or change the generated music based on input from the one or moresensors to guide the user toward the desired location.

Additionally and/or alternatively, in a two-parameter model, one or moresensors may be used to detect a user's position relative to a desiredlocation and a user's direction of movement. As a user moves, thesensors may detect (1) whether the user is moving closer to a desiredsecond location or away from a desired second location and (2) thedirection (angle relative to the desired location) the user is moving.Based on the input from the sensors, the computing system may thengenerate music indicating (1) that the user is moving closer to or awayfrom the desired second location and/or (2) the direction the user ismoving relative to the object. In a non-limiting example, the rhythm ofthe music/audio/sound may increase as the user gets closer to thedesired location while the music/audio/sound becomes more random if theuser is heading in the wrong direction. The computing system maycontinuously control and/or change the generated music/audio/sound basedon input from the one or more sensors to guide the user toward thedesired location.

Music attributes 915 a-915 n may further vary based on the type ofapplication/communication (e.g., voice, text, speech, fitness tracker,EEG device, smart watch, AR/VR device, facial recognition device,camera, user input, and/or the like), the demographics (e.g., age, sex,and/or the like) of the user, the user's preferences, input from a user,and/or the like.

FIG. 9C represents a sysyem 900 c having three musical/sound/audiocharacteristics: (1) amplitude (volume of music/audio/sound), (2) attackrate (rhythm, arpeggiation), and (3) note probabilities (how likely aparticular note will be played). State 1 (current state) has threecorresponding values for amplitude, attack rate, and note probabilitiesand state 2 (goal state) has three different corresponding values foramplitude, attack rate, and note probabilities. One or more sensors maybe used to measure where a user is i.e. state 1, state 2, or betweenstates 1 and 2. Based on the measured state of the user,music/audio/sound may be continuously generated and controlled to lead auser from state 1 to state 2. In a non-limiting example, movementdirection may control attack rate and note probabilities while distancefrom goal may control amplitude.

FIG. 9D represents an XY coordinate system 900 d for mapping differentstates to a region on an XY coordinate system. This XY coordinate system900 d may be used in place of or in addition to user interface 200 shownin FIG. 2 and function in a similar way as user interface 200 to controlone or more musical parameters. The position on a two ormore-dimensional graph can be used as coordinates to drive a multi-inputmusic system where different axes control different musical attributes.The X-axis might correspond to one or more characteristics ofmusic/audio/sound while the Y-axis might correspond to one or moredifferent characteristics of music/audio/sound. In a non-limitingexample, The X-axis might correspond to a valence of music/audio/sound,i.e. positive/minor notes, randomness/synchronicity of notes, and/or thelike, while the Y-axis might correspond to intensity ofmusic/audio/sound, i.e. loudness, number of beats per minute, and/or thelike.

Each state may be mapped to a region on the XY coordinate system shownin FIG. 9D. The XY coordinate system is not limited to only being a XYplane. The XY plane could be any pattern/shape (e.g., an oval, circle,triangle, square, rectangle, grid, XYZ coordinate system, 2-D model, 3-Dmodel, etc.). A person of ordinary skill in the art would understandthat any pattern may act in a similar manner as the XY plane describedbelow.

The XY plane 900 d may be displayed to a user on a computing systemand/or communication device. Additionally and/or alternatively, thecomputing system and/or communication device may use the XY plane todetermine one or more musical parameters corresponding to a user'sdetermined state.

A user may interact with XY plane 900 d via tactile input and/or mouseinput. Additionally and/or alternatively, the XY plane 900 d may be usedby a computing system and/or user device to determine what music to playbased on a state contained within a communication (e.g., at least one ofa sensor communication, an IoT communication, a biometric/healthcommunication, a voice communication, a textual communication, aphotographic communication, a video communication, and/or the like). Acomputing system may determine where a particular state is mapped on XYplane 900 d and access and play the algorithm associated with theparticular state based on where the state is positioned on the XY plane900 d.

Based on the user interaction with the XY plane 900 d and/or based on adetermined state from a communication, the computing system maydetermine the distance from an x-axis on the XY plane 900 d associatedwith the determined at least one particular state and/or a distance froma y-axis on the XY plane 900 d. Based on the determination of theposition (distance from x-axis and/or y-axis) associated with thedetermined at least one state, the computing system and/or user devicemay generate music having the valence and the intensity associated withthe determined at least one state of the communication.

Additionally and/or alternatively, the XY plane 900 d may facilitatetransitioning between music associated with a first particular state tomusic associated with a second particular state. In FIG. 9D, stateslocated in different regions of XY plane 900 d. A user may firstindicate a first state (state 1) via touch input, mouse input, and/or byindicating a state in a communication.

If the selection is made via touch or a mouse and after a first userindication is made, then a computing system and/or communication devicemay track a user's interaction with the XY plane 900 d. A user maycontinue to drag his or her finger or other device (e.g., mouse, stylus,and/or the like) along a path on the XY plane 900 d indicating that themusic that is generated should transition between music associated withat least two states. The music that is generated may transition in realtime or near real time based on the user's interaction with the XY plane900 d. For example, if the user stops for a period of time on aparticular region associated with a state, then the music associatedwith that particular state may play for that period of time until theuser stops selecting the region and/or moves on to a different regionassociated with a different state. Additionally and/or alternatively,each state that the user selects by dragging his or her finger or otherdevice (e.g., mouse, stylus, and/or the like) along a XY plane 900 d mayplay music associated with that particular state for a predeterminedamount of time (e.g., until a note is finished playing, until a sequenceis complete, and/or the like) before transitioning on to the musicassociated with the next selected state/region. In some cases, thepredetermined amount of time may be selected by a user of XY plane 900d.

The computing system and/or user device may also introduce some lag sothat music that is generated by user interaction is not generated inreal time. For example, if the user drags his or her finger or deviceover XY plane 900 d quickly, the computing system may not be able totransition between music associated with different states smoothly. Byintroducing lag, the computing system may smoothly transition betweenmusic associated with different emotions.

Additionally and/or alternatively, if a user picks up his finger and/ordevice, a computer may determine that a user would like to pause betweentransitioning between a first selected state and an additional selectedstate. The music may turn off or continue to play in static position. Ifthe user rests on a particular state, the music can still becontinuously generated.

If a selection of more than one state is made via a communication (e.g.,at least one of a sensor communication, an IoT communication, abiometric/health communication, a voice communication, a textualcommunication, a photographic communication, a video communication,and/or the like) and the communication contains at least two states,then a computing device may determine a path on XY plane 900 d between afirst selected state and at least one additional state. Using the statesbetween the first selected state and at least one additional state, thecomputing system may smoothly transition between playing musicassociated with each of the at least two states contained within thecommunication by playing music associated with the determined additionalstates between music associated with the at least two particular statesindicated in the communication.

Additionally and/or alternatively, the generated music may transitionbetween music associated with the at least two states indicated in thecommunication by pausing music associated with a first indicated statebefore playing music associated with a second indicated state.

In a non-limiting example, a computing system may determine that theuser's goal state is state 2. In order to determine a user's goal state,a user may select an option for a desired state (e.g., work-out option,navigation option, etc.) on a user interface (e.g., user interface 200,interface 900 d, or other interface, etc.). Additionally and/oralternatively, a computing system may receive feedback from one or moresensors (e.g., an indication of a high stress state, etc.) and determinea user's goal state (e.g., calm, relaxed, etc.) based on the feedbackfrom the one or more sensors. In some cases, a user may also select alength of time or a distance (e.g., a mile, a kilometer, etc.) to runthe music generation application.

Based on one or more sensor inputs, the computing system may determinethat the user is currently at state 1, between state 1 and 2, how closea user is to state 2, and/or the like. The computing system mightcontinuously control synthesized music/audio/sound to guide the user tostate 2 based on the one or more sensor inputs and/or indicate to a userhow far away or close the user is to state 2. As the user gets closer tostate 2, the music will adapt to incorporate more elements of state 2.As the user gets further from state 2 the music will adapt toincorporate more elements of state 1. Thus, a user may receivecontinuous feedback about how close he or she is to state 1 or state 2.

FIG. 9E represents a schematic diagram 900 e for receiving multipleinputs from different sources to generate music associated with one ormore states, in accordance with various embodiments. The music that isgenerated may be influenced by a passive control (e.g.,sensor/communication input) and/or an active control (e.g., a userinterface). Additionally and/or alternatively, the music that isgenerated may be based on input from two passive controls (e.g., twosensors, etc.) and/or two active controls (e.g., two user interfaces,two inputs via a user interface, etc.) and the music/sound that isgenerated from the two passive controls and/or two active controls maybe generated based on a similar method described below.

In various embodiments, a first source 920 might be received from one ormore passive source(s) (e.g., one or more sensors and/or communications(e.g., an IoT communication, a biometric/health communication, a voicecommunication, a textual communication, a picture communication, a videocommunication, a haptic communication, etc.)). Data received from thepassive source 920 may be continuously received, received periodically(e.g., every second, every minute, every hour, and/or the like), and/orreceived once. Data received from the passive source 920 may beprocessed through a data processing algorithm 925 to determine a stateof a user and/or an environment. The determined state of the user and/orenvironment may be mapped to a location as shown in FIG. 9D and/or FIG.2, given a binary value, and/or given a number between 0 and 1 (930).The location, binary value, and/or number may control one or moremusical parameters and cause a computing system to generate music basedon the one or more audio feature parameter values (935). The music thatis generated may be continuously updated/controlled by the continuousone or more active or passive inputs, regularly updated/controlled basedon inputs received periodically, updated only when a discrete message isreceived, and/or the like.

In some cases, a second source 940 may be added to control thegeneration of music. The second source 940 may be an active source whichreceives direct user input via a user interface (e.g., user interface200, user interface 900 d), a touch screen, one or more actuatedbuttons, and/or the like. A user may control one or more of turning themusic off and on, a genre of music created, one or more musicalinstruments being played, transitioning between different types ofmusic, selecting different states, selecting a goal state, and/or thelike.

The music generated from the inputs (e.g., first source 920 and secondsource 940) may be harmonized, layered, cross-faded, and/or the like.For example, the music may start out generating a user's current statebased on sensor input and based on a selection from a user via a userinterface of a goal state, the music that is generated may thentransition/change to guide a user from a first state to a second state.Additionally and/or alternatively, in another non-limiting example, themusic that is generated based on sensor input may have one or moremusical attributes and a user selection of age, genre, etc. may causethe music to adapt or change based on the selections. Additionallyand/or alternatively, the music generated based on one or more sensorsmay adapt or change as a user drags his or her finger over a userinterface (e.g., user interface 200 and/or 900 d).

In another non-limiting example, sensor input may be received from twopassive sources e.g., two biometric inputs, a biometric input and acamera, etc. The sensor input may be associated with two different usersand/or participants in a race or competitive activity. The music that isgenerated may be used to reflect a user's position in the race orcompetitive activity. The music may speed up or slow down to reflect ifthe user is ahead or behind other participants in the race orcompetitive activity.

These are only some examples of the different ways music may begenerated using two different input sources.

Although FIGS. 9A-9E are directed toward guiding a user from a firststate to a second state, similar methods may be used to exercisecontinuous control over music generation to guide a user toward otherparticular desired states or goals. Additionally and/or alternatively,similar methods may be used to transition between environmental states(e.g., transitioning from sunny to rainy weather, transitioning from onetemperature to another temperature, and/or the like). Additionallyand/or alternatively, the user interface described with respect to FIGS.2-4 may be used to exercise continuous control music generation to guidea user from a first state to a second desired state.

For example, systems similar to those described in FIGS. 2-4 and 9 maybe used to continuously control the generation of music to guide a userfrom a stressful/aroused state to a relaxed/calm state, from anon-meditative state to a meditative state, from an unfocused state to afocused state, from a negative state to a positive state, from a stateof inactivity to a state of activity, from a slow breathing pattern to afast breathing pattern, from a fast breathing pattern to a slowbreathing pattern, from a slow heartbeat to a fast heartbeat, from aslow heartbeat to a fast heartbeat, from bad posture to good posture,from less movement to more movement and vice versa, from a bad bodyposition to a good body position, and/or the like. Additionally and/oralternatively, systems similar to those described in FIGS. 2-4 and 9 maybe used to continuously control the generation of music to guide a userthrough a workout (warm-up, exercise, cool-down, etc.).

Additionally and/or alternatively, systems similar to those described inFIGS. 2-4 and 9 may be used to continuously control the generation ofmusic to sync to at least one of a running speed of the user, aheartbeat of the user, a breathing pattern of the user, a brainwavepattern of the user, and/or the like. In this manner, a user may receiveinstant feedback about his or her current state.

In various embodiments, these systems and methods may be used tocontinuously generate music that reflects a user's current state and maybe constantly adapted/changed in real-time as the user's state changes.

FIG. 10 is a flow diagram illustrating a method for continuouslycontrolling the generation of the music to guide a user toward a goal,in accordance with various embodiments.

While the techniques and procedures are depicted and/or described in acertain order for purposes of illustration, it should be appreciatedthat certain procedures may be reordered and/or omitted within the scopeof various embodiments. Moreover, while the method illustrated by FIG.10 can be implemented by or with (and, in some cases, are describedbelow with respect to) the system 100 of FIG. 1 (or components thereof)the user interface 200 of FIG. 2 (or components thereof), the system 300of FIG. 3 (or components thereof), the mapping system 400 of FIG. 4 (orcomponents thereof), and/or the mapping system of FIG. 9 (or componentsthereof), such methods may also be implemented using any suitablehardware (or software) implementation. Similarly, while each of thesystem 100 of FIG. 1 (or components thereof) user interface 200 of FIG.2 (or components thereof), the system 300 of FIG. 3 (or componentsthereof), the mapping system 400 of FIG. 4 (or components thereof),and/or the mapping system of FIG. 9 (or components thereof), can operateaccording to the method illustrated by FIG. 10 (e.g., by executinginstructions embodied on a computer readable medium), the system 100 ofFIG. 1 user interface 200 of FIG. 2, the system 300 of FIG. 3, themapping system 400 of FIG. 4, and/or the mapping system of FIG. 9 caneach also operate according to other modes of operation and/or performother suitable procedures.

In the non-limiting embodiment of FIG. 10, the method, at block 1005,might comprise receiving, with a computing system, at least one sensorinput associated with a user and/or an environment.

The computing system may be at least one of a desktop computer, a laptopcomputer, a tablet, a smart phone, an e-reader, and/or the like.Additionally and/or alternatively, in some embodiments, the computingsystem might include, without limitation, one of a processor of aset-top box, a processor of a digital video recording (“DVR”) device, aprocessor of a user device running a software application (“app”), aprocessor of an audio playback device, a processor on an input device(e.g., fitness tracker, EEG device, or the like) running an app, aprocessor of a media player, a processor of a gaming console, aprocessor in audio equipment, and/or the like.

The sensor input may contain feedback from one or more sensorsincluding, but not limited to, one or more GPS sensors, one or moredistance sensors, one or more motion sensors, one or more movementsensors, one or more speed or velocity sensors, one or moreaccelerometer sensors, one or more eye tracking sensors, one or morebiometric/health sensors, one or more facial recognition sensors, one ormore camera sensors, and/or the like.

The biometric/health sensor input may be received from at least one of afitness tracker, a smart watch, a smart phone, an electroencephalography(“EEG”) device, a virtual reality (“VR”) device, an augmented reality(“AR”) device, and/or the like. The feedback from the one or morebiometric/health sensors might include, but is not limited to, at leastone of a heart rate, a blood pressure, a stress level, a measure ofelectrical activity within a brain, pupil dilation, skin conductivity, alevel of activity, number of steps, and/or the like.

The method of FIG. 10, at block 1010, may further comprise analyzing,with the computing system, the at least one sensor input to determine atleast one first state of the user. The state of the user may correspondto at least one of an emotion of a user, a feeling of a user, a locationof the user, a physical position (e.g., a posture of a user, a bodyposition of a user, etc.) of a user, a level of activity of a user, adirection of an action of a user, an action (e.g., walking, running,biking, etc.) of a user, and/or the like.

In various embodiments, the method of FIG. 10, at block 1015, mayadditionally include determining, with the computing system, a desiredsecond state of the user. The desired second state of a user maycorrespond to at least one of an emotion of a user, a feeling of a user,a location of the user, a physical position (e.g., a posture of a user,a body position of a user, etc.) of a user, a level of activity of auser, an action (e.g., walking, running, biking, etc.) of a user, and/orthe like.

A user may manually enter a desired second state. Additionally and/oralternatively, the computing system may determine a desired secondstate. In a non-limiting example, the computing system, using the one ormore sensors, may detect that a user is feeling stressed or anxious.Based on the determination that a user is stressed or anxious, thecomputing system may determine that the desired second state is calm orrelaxed.

At block 1020, the method of FIG. 10 may determine whether the at leastone first state of the user matches the desired second state of theuser. Based on a determination that the at least one first state of theuser does not match the desired second state of the user, the method1000, at block 1025 may continuously control, with the computing system,the generation of music having one or more characteristics to guide theuser to the desired second state based on the at least at least onesensor input. This process may continue as indicated by arrow 1030 untilthe computing system determines that the user has achieved the desiredsecond state.

The computing system may be configured to autonomously determine andcontinuously generate one or more characteristics of music to guide auser toward a desired second state based on the sensor feedback. Thegenerated music may be designed to gradually lead a user in astep-by-step process toward the desired second state. In other words,the generated music may continuously adapt and/or evolve based onfeedback received from the one or more sensors to lead a user toward adesired second state.

In order to guide the user toward a desired second state, the computingsystem may control, adapt, and/or evolve one or more characteristics ofmusic based on input from the one or more sensors. The one or morecharacteristics of music might, include, but are not limited to, a notepattern, a harmony, a tone, vocal-like qualities, elements of surprise,randomization, consistency, crescendo, decrescendo an orchestration, aspeed, a rhythm, a beat, or a volume. The one or more characteristics ofmusic might be associated with guiding a user toward a desired state. Ina non-limiting example, a beat or rhythm might be designed to help auser control his or her breathing. The one or more first characteristicsof a plurality of characteristics of music may further be associatedwith the at least one of the age, the sex or other demographic of auser.

The following examples represent ways the generated music may be used toguide a user from a first state to a desired second. These are examplesonly and they are not intended to limit the scope of the invention.

In a first non-limiting example, the generated music may be designed toguide the user from a first stressful state to a second more relaxedstate based on the at least one sensor input. The computing system maydetermine that a user is stressed based on input from the sensor andgenerate music to soothe and relax the user. The music that is generatedmay be designed to gradually lead the user to a more relaxed state.Initially, the music may start out louder and faster and as thecomputing system determines that the user is calming down (based onsensor input) the music may become softer and slower until the desiredsecond state is reached. Once the desired second state is reached thecomputing system may generate music that is consistent with the desiredsecond state.

In a second non-limiting example, the generated music may be designed toguide the user from a first non-meditative state or stressed to a secondmeditative or relaxed state based on the at least one sensor input. Themusic that is generated may be designed to gradually lead the user to ameditative or relaxed state. Initially, the music may start out louderand brighter in timbre and as the computing system determines that theuser is entering a meditative or relaxed state (based on the sensorinput) the music may become darker in timbre and slower until thedesired second state is reached. Once the desired second state isreached the computing system may generate music that is consistent withthe meditative or relaxed state.

In a third non-limiting example, the generated music might be designedto guide the user from a first unfocused state to a second focused statebased on the at least one sensor input.

In a fourth non-limiting example, the generated music might be designedto guide the user from a first negative emotion to a second positiveemotion.

In a fifth non-limiting example, the generated music might be designedto guide the user through a periodic activity (e.g., a desired number ofsteps per minute, a desired number of repetitions per minute, and/or thelike) based on input from the one or more sensors. The music that isgenerated may be designed to gradually lead the user through theperiodic activity (e.g., toward a desired number of steps per minute,toward a desired number of repetitions per minute, and/or the like).Initially, the music may start out with a slower beat. Once thecomputing system determines that the user's periodic activity matchesthe slower beat (based on input from the sensor), the computing systemmight increase the beat until the user is at the desired periodicactivity. Once the desired periodic activity is reached the computingsystem may generate music that is consistent with maintaining thedesired periodic activity.

In a sixth non-limiting example, the generated music might be designedto sync to at least one of a running speed of the user, a heartbeat ofthe user, a breathing pattern of the user, a brainwave pattern of theuser, and/or the like based on input from the one or more sensors.

In a seventh non-limiting example, the generated music might be designedto guide the user to at least one of a slower breathing pattern, aslower heartbeat, a faster breathing pattern, a faster heartrate, and/orthe like based on input from the one or more sensors.

In an eighth non-limiting example, the generated music may be designedto lead a user through a workout routine. The music may start out slowand gradually speed up (based on input from the one or more sensors) tolead a user through a warm-up routine. The computing system maydetermine, based on biometric/health feedback from the user, when theuser is ready to move on to the next stage of a workout routine andprovide cues to the user that it is time to move on through thegenerated music. Additionally and/or alternatively, a user may indicatethat he or she would like to exercise for an hour. The computing systemmay then lead a user through an hour exercise routine (warm-up,cool-down, etc.) using the generated music and the input from the one ormore sensors.

In a ninth non-limiting example, the generated music may be designed toguide the user from a first location to a second desired location. Thisfeature may be used by a user who is visually impaired to guide to adesired second location. For example, a user may indicate that he or shewould like to go to the kitchen. Using sensor input, the computingsystem may generate music to guide the user towards the kitchen. Themusic may become faster as the user gets closer to the kitchen or themusic may become slower as the user moves away from the kitchen. Thesensor input may be used to continuously update and control thegenerated music.

In a tenth non-limiting example, the generated music may be designed toguide the user to at least one of a better posture, a better bodyposition, and/or through a physical therapy routine. Sensor input may beused to provide information about a user's posture or body informationand different musical characteristics may be designed to guide the userto better posture, body position, or toward a correct position forphysical therapy.

In an eleventh non-limiting example, the generated music may be designedto alleviate pain and/or reflect a user's current state of pain. Sensorinput may be used to detect a user's current level of pain and differentmusical attributes may be used to guide a user to a less painful state,distract a user from a painful state, and/or provide a doctor withfeedback about a patient's current state of pain.

In a twelfth non-limiting example, the generated music might be designedto help a user win a race or determine where a user is within a race.Sensor input may be received from one or more passive sources e.g., twobiometric inputs, a biometric input and a camera, etc. The sensor inputmay be associated with two different users and/or participants in arace. The music that is generated may be used to reflect a user'sposition in a race. The music may speed up or slow down to reflect ifthe user is ahead or behind other participants in a race.

The examples above of guiding a user toward a desired second state areintended to be non-limiting. Generating music using sensors may be usedto guide users towards many other states than those listed above.

An exemplary system and hardware implementation will now be described,with reference to FIGS. 11 and 12.

FIG. 11 is a block diagram illustrating an exemplary computer or systemhardware architecture, in accordance with various embodiments. FIG. 11provides a schematic illustration of one embodiment of a computer system1100 of the service provider system hardware that can perform themethods provided by various other embodiments, as described herein,and/or can perform the functions of computer or hardware system (i.e.,computing systems/user device 105, input devices 115, audio playbackdevices 120 a-120 n, music sources (or servers) 125, user interface 200,computing system 305, input devices 310, XY interface 900 d, etc.), asdescribed above. It should be noted that FIG. 11 is meant only toprovide a generalized illustration of various components, of which oneor more (or none) of each may be utilized as appropriate. FIG. 11,therefore, broadly illustrates how individual system elements may beimplemented in a relatively separated or relatively more integratedmanner.

The computer or hardware system 1100—which might represent an embodimentof the computer or hardware system (i.e., computing systems/user devices105, input devices 115, audio playback devices 120 a-120 n, musicsources (or servers) 125, user interface 200, computing system 305,input devices 310, XY interface 900 d, etc.), described above withrespect to FIGS. 1-10—is shown comprising hardware elements that can beelectrically coupled via a bus 1105 (or may otherwise be incommunication, as appropriate). The hardware elements may include one ormore processors 1110, including, without limitation, one or moregeneral-purpose processors and/or one or more special-purpose processors(such as microprocessors, digital signal processing chips, graphicsacceleration processors, and/or the like); one or more input devices1115 (i.e., input devices 115, input devices 310, etc.), which caninclude, without limitation, a mouse, a keyboard, fitness trackers,smart watches, EEG devices, and/or the like; and one or more outputdevices 1120, which can include, without limitation, a display device, aprinter, and/or the like.

The computer or hardware system 1100 may further include (and/or be incommunication with) one or more storage devices 1125, which cancomprise, without limitation, local and/or network accessible storage,and/or can include, without limitation, a disk drive, a drive array, anoptical storage device, solid-state storage device such as a randomaccess memory (“RAM”) and/or a read-only memory (“ROM”), which can beprogrammable, flash-updateable and/or the like. Such storage devices maybe configured to implement any appropriate data stores, including,without limitation, various file systems, database structures, musicalalgorithms associated with different emotions, and/or the like.

The computer or hardware system 1100 might also include a communicationssubsystem 1130, which can include, without limitation, a modem, anetwork card (wireless or wired), an infra-red communication device, awireless communication device and/or chipset (such as a Bluetooth™device, an 802.11 device, a WiFi device, a WiMax device, a WWAN device,cellular communication facilities, etc.), and/or the like. Thecommunications subsystem 1130 may permit data to be exchanged with anetwork (such as the network described below, to name one example), withother computer or hardware systems, and/or with any other devicesdescribed herein. In many embodiments, the computer or hardware system1100 will further comprise a working memory 1135, which can include aRAM or ROM device, as described above.

The computer or hardware system 1100 also may comprise softwareelements, shown as being currently located within the working memory1135, including an operating system 1140, device drivers, executablelibraries, and/or other code, such as one or more application programs1145, which may comprise computer programs provided by variousembodiments (including, without limitation, hypervisors, VMs, and thelike), and/or may be designed to implement methods, and/or configuresystems, provided by other embodiments, as described herein. Merely byway of example, one or more procedures described with respect to themethod(s) discussed above might be implemented as code and/orinstructions executable by a computer (and/or a processor within acomputer); in an aspect, then, such code and/or instructions can be usedto configure and/or adapt a general purpose computer (or other device)to perform one or more operations in accordance with the describedmethods.

A set of these instructions and/or code might be encoded and/or storedon a non-transitory computer readable storage medium, such as thestorage device(s) 1125 described above. In some cases, the storagemedium might be incorporated within a computer system, such as thesystem 1100. In other embodiments, the storage medium might be separatefrom a computer system (i.e., a removable medium, such as a compactdisc, etc.), and/or provided in an installation package, such that thestorage medium can be used to program, configure and/or adapt a generalpurpose computer with the instructions/code stored thereon. Theseinstructions might take the form of executable code, which is executableby the computer or hardware system 1100 and/or might take the form ofsource and/or installable code, which, upon compilation and/orinstallation on the computer or hardware system 1100 (e.g., using any ofa variety of generally available compilers, installation programs,compression/decompression utilities, etc.) then takes the form ofexecutable code.

It will be apparent to those skilled in the art that substantialvariations may be made in accordance with specific requirements. Forexample, customized hardware (such as programmable logic controllers,field-programmable gate arrays, application-specific integratedcircuits, and/or the like) might also be used, and/or particularelements might be implemented in hardware, software (including portablesoftware, such as applets, etc.), or both. Further, connection to othercomputing devices such as network input/output devices may be employed.

As mentioned above, in one aspect, some embodiments may employ acomputer or hardware system (such as the computer or hardware system1100) to perform methods in accordance with various embodiments of theinvention. According to a set of embodiments, some or all of theprocedures of such methods are performed by the computer or hardwaresystem 1100 in response to processor 1110 executing one or moresequences of one or more instructions (which might be incorporated intothe operating system 1140 and/or other code, such as an applicationprogram 1145) contained in the working memory 1135. Such instructionsmay be read into the working memory 1135 from another computer readablemedium, such as one or more of the storage device(s) 1125. Merely by wayof example, execution of the sequences of instructions contained in theworking memory 1135 might cause the processor(s) 1110 to perform one ormore procedures of the methods described herein.

The terms “machine readable medium” and “computer readable medium,” asused herein, refer to any medium that participates in providing datathat causes a machine to operate in a specific fashion. In an embodimentimplemented using the computer or hardware system 1100, various computerreadable media might be involved in providing instructions/code toprocessor(s) 1110 for execution and/or might be used to store and/orcarry such instructions/code (e.g., as signals). In manyimplementations, a computer readable medium is a non-transitory,physical, and/or tangible storage medium. In some embodiments, acomputer readable medium may take many forms, including, but not limitedto, non-volatile media, volatile media, or the like. Non-volatile mediaincludes, for example, optical and/or magnetic disks, such as thestorage device(s) 1125. Volatile media includes, without limitation,dynamic memory, such as the working memory 1135. In some alternativeembodiments, a computer readable medium may take the form oftransmission media, which includes, without limitation, coaxial cables,copper wire and fiber optics, including the wires that comprise the bus1105, as well as the various components of the communication subsystem1130 (and/or the media by which the communications subsystem 1130provides communication with other devices). In an alternative set ofembodiments, transmission media can also take the form of waves(including without limitation radio, acoustic and/or light waves, suchas those generated during radio-wave and infra-red data communications).

Common forms of physical and/or tangible computer readable mediainclude, for example, a floppy disk, a flexible disk, a hard disk,magnetic tape, or any other magnetic medium, a CD-ROM, any other opticalmedium, punch cards, paper tape, any other physical medium with patternsof holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chipor cartridge, a carrier wave as described hereinafter, or any othermedium from which a computer can read instructions and/or code.

Various forms of computer readable media may be involved in carrying oneor more sequences of one or more instructions to the processor(s) 1110for execution. Merely by way of example, the instructions may initiallybe carried on a magnetic disk and/or optical disc of a remote computer.A remote computer might load the instructions into its dynamic memoryand send the instructions as signals over a transmission medium to bereceived and/or executed by the computer or hardware system 1100. Thesesignals, which might be in the form of electromagnetic signals, acousticsignals, optical signals, and/or the like, are all examples of carrierwaves on which instructions can be encoded, in accordance with variousembodiments of the invention.

The communications subsystem 1130 (and/or components thereof) generallywill receive the signals, and the bus 1105 then might carry the signals(and/or the data, instructions, etc. carried by the signals) to theworking memory 1135, from which the processor(s) 1110 retrieves andexecutes the instructions. The instructions received by the workingmemory 1135 may optionally be stored on a storage device 1125 eitherbefore or after execution by the processor(s) 1110.

Additionally and/or alternatively, system 1100 may utilize neuralnetworks (e.g., convolutional, recurrent, spiking, or other statisticallearning mode) to classify an emotional state, determine emotion statefrom incoming sensor data, control position in the user interface, etc.

As noted above, a set of embodiments comprises methods, systems, andapparatuses for generating music, and, more particularly, to methods,systems, and apparatuses for generating music associated with a state oran emotion contained within a communication, for a user interface forgenerating music associated with a state or an emotion, and forgenerating music to guide a user toward a desired state or goal.

FIG. 12 illustrates a schematic diagram of a system 1200 that can beused in accordance with one set of embodiments. The system 1200 caninclude one or more user computers, user devices, or customer devices1205 (similar to computing systems/user devices 105, input devices 115,etc.). A user computer, user device, or customer device 1205 can be ageneral purpose personal computer (including, merely by way of example,desktop computers, tablet computers, laptop computers, handheldcomputers, and the like, running any appropriate operating system,several of which are available from vendors such as Apple, MicrosoftCorp., and the like), cloud computing devices, a server(s), and/or aworkstation computer(s) running any of a variety ofcommercially-available UNIX™ or UNIX-like operating systems. A usercomputer, user device, or customer device 1205 can also have any of avariety of applications, including one or more applications configuredto perform methods provided by various embodiments (as described above,for example), as well as one or more office applications, databaseclient and/or server applications, and/or web browser applications.Alternatively, a user computer, user device, or customer device 1205 canbe any other electronic device, such as a thin-client computer,Internet-enabled mobile telephone, and/or personal digital assistant,capable of communicating via a network (e.g., the network(s) 1210described below) and/or of displaying and navigating web pages or othertypes of electronic documents. Although the exemplary system 1200 isshown with three user computers, user devices, or customer devices 1205,any number of user computers, user devices, or customer devices can besupported.

Certain embodiments operate in a networked environment, which caninclude a network(s) 1210. The network(s) 1210 can be any type ofnetwork familiar to those skilled in the art that can support datacommunications using any of a variety of commercially-available (and/orfree or proprietary) protocols, including, without limitation, TCP/IP,SNA™, IPX™, AppleTalk™, and the like. Merely by way of example, thenetwork(s) 1210 (similar to network(s) 140 FIG. 1, or the like) can eachinclude a local area network (“LAN”), including, without limitation, afiber network, an Ethernet network, a Token-Ring™ network and/or thelike; a wide-area network (“WAN”); a wireless wide area network(“WWAN”); a virtual network, such as a virtual private network (“VPN”);the Internet; an intranet; an extranet; a public switched telephonenetwork (“PSTN”); an infra-red network; a wireless network, including,without limitation, a network operating under any of the IEEE 702.11suite of protocols, the Bluetooth™ protocol known in the art, and/or anyother wireless protocol; and/or any combination of these and/or othernetworks. In a particular embodiment, the network might include anaccess network of the service provider (e.g., an Internet serviceprovider (“ISP”)). In another embodiment, the network might include acore network of the service provider, and/or the Internet.

Embodiments can also include one or more server computers 1215. Each ofthe server computers 1215 may be configured with an operating system,including, without limitation, any of those discussed above, as well asany commercially (or freely) available server operating systems. Each ofthe servers 1215 may also be running one or more applications, which canbe configured to provide services to one or more clients 1205 and/orother servers 1215.

Merely by way of example, one of the servers 1215 might be a dataserver, a web server, a cloud computing device(s), or the like, asdescribed above. The data server might include (or be in communicationwith) a web server, which can be used, merely by way of example, toprocess requests for web pages or other electronic documents from usercomputers 1205. The web server can also run a variety of serverapplications, including HTTP servers, FTP servers, CGI servers, databaseservers, Java servers, and the like. In some embodiments of theinvention, the web server may be configured to serve web pages that canbe operated within a web browser on one or more of the user computers1205 to perform methods of the invention. A user interface (similar touser interface 200) may be featured on a webpage hosted by one of theservers 1215.

The server computers 1215, in some embodiments, might include one ormore application servers, which can be configured with one or moreapplications accessible by a client running on one or more of the usercomputers 1205 and/or other servers 1215. In a non-limiting example, auser interface (similar to user interface 200) may be run as anapplication by a client running on one or more of the user computers1205 and/or other servers 1215. Merely by way of example, the server(s)1215 can be one or more general purpose computers capable of executingprograms or scripts in response to the user computers 1205 and/or otherservers 1215, including, without limitation, web applications (whichmight, in some cases, be configured to perform methods provided byvarious embodiments). Merely by way of example, a web application can beimplemented as one or more scripts or programs written in any suitableprogramming language, such as Java™, C, C#™ or C++, and/or any scriptinglanguage, such as Perl, Python, or TCL, as well as combinations of anyprogramming and/or scripting languages. The application server(s) canalso include database servers, including, without limitation, thosecommercially available from Oracle™, Microsoft™, Sybase™ IBM™, and thelike, which can process requests from clients (including, depending onthe configuration, dedicated database clients, API clients, webbrowsers, etc.) running on a user computer, user device, or customerdevice 1205 and/or another server 1215. In some embodiments, anapplication server can perform one or more of the processes forgenerating music, and, more particularly, for generating musicassociated with a state or an emotion contained within a communicationand for a user interface for generating music associated with a state oran emotion. Data provided by an application server may be formatted asone or more web pages (comprising HTML, JavaScript, etc., for example)and/or may be forwarded to a user computer 1205 via a web server (asdescribed above, for example). Similarly, a web server might receive webpage requests and/or input data from a user computer 1205 and/or forwardthe web page requests and/or input data to an application server. Insome cases, a web server may be integrated with an application server.

Additionally and/or alternatively, user computer 1205 may utilize neuralnetworks (e.g., convolutional, recurrent, spiking, or other statisticallearning mode) to classify a state or an emotional state, determine astate or an emotional state from incoming sensor data, control positionin the user interface, etc.

In accordance with further embodiments, one or more servers 1215 canfunction as a file server and/or can include one or more of the files(e.g., application code, data files, etc.) necessary to implementvarious disclosed methods, incorporated by an application running on auser computer 1205 and/or another server 1215. Alternatively, as thoseskilled in the art will appreciate, a file server can include allnecessary files, allowing such an application to be invoked remotely bya user computer, user device, or customer device 1205 and/or server1215.

It should be noted that the functions described with respect to variousservers herein (e.g., application server, database server, web server,file server, etc.) can be performed by a single server and/or aplurality of specialized servers, depending on implementation-specificneeds and parameters.

In certain embodiments, the system can include one or more databases1220 a-1220 n (collectively, “databases 1220”). The location of each ofthe databases 1220 is discretionary: merely by way of example, adatabase 1220 a might reside on a storage medium local to (and/orresident in) a server 1215 a (and/or a user computer, user device, orcustomer device 1205). Alternatively, a database 1220 n can be remotefrom any or all of the computers 1205, 1215, so long as it can be incommunication (e.g., via the network 1210) with one or more of these. Ina particular set of embodiments, a database 1220 can reside in astorage-area network (“SAN”) familiar to those skilled in the art.(Likewise, any necessary files for performing the functions attributedto the computers 1205, 1215 can be stored locally on the respectivecomputer and/or remotely, as appropriate.) In one set of embodiments,the database 1220 can be a relational database, such as an Oracledatabase, that is adapted to store, update, and retrieve data inresponse to SQL-formatted commands. The database might be controlledand/or maintained by a database server, as described above, for example.

According to some embodiments, system 1200 might further comprise one ormore input devices 1230 (similar to input devices 115 of FIG. 1, or thelike), one or more audio playback devices 1240 a-1240 n (similar toaudio playback devices 120 a-120 n of FIG. 1, or the like), one or moremusic (e.g., video) content sources 1245 (similar to music sources (orservers) 125, or the like) and corresponding database(s) 1250 (similardatabases 130 of FIG. 1, or the like), and/or the like. In someembodiments, the computing system 1200 might be communicatively coupledto one or more input device, one or more a playback device(s), or thelike (i.e., one or more of input devices 115 or 1230 and/or audioplayback device(s) 120 a-120 n or 1240 a-1240 n, or the like). In somecases, the input device might comprise one of a fitness tracker, EEGdevice, smart watch, and/or the like.

These and other functions of the system 1200 (and its components) aredescribed in greater detail above with respect to FIGS. 1-11.

In one aspect, a method for creating generative music might becharacterized by the following numbered paragraphs:

1. A method, comprising: analyzing, with a computing system, acommunication to determine at least one state contained within thecommunication; autonomously determining, with the computing system, oneor more first characteristics of a plurality of characteristics of musicassociated with the determined at least one state contained within thecommunication; and based on the determination of the one or more firstcharacteristics of a plurality of characteristics of music associatedwith the at least one state contained within the communication,autonomously generating, with the computing system, music having the oneor more first characteristics of the plurality of characteristicsassociated with the at least one determined state contained within thecommunication.

2. The method of paragraph 1, wherein the computing system comprises atleast one of a desktop computer, a laptop computer, a tablet, anembedded processing unit, or a smart phone.

3. The method of paragraph 1, wherein the one or more firstcharacteristics of the plurality of characteristics of music include atleast one of a pitch, note pattern, note envelope or shape, a controlsignal, a harmony, a grouping, vocal-like filtering and inflections,elements of surprise, randomization, consistency, crescendo, decrescendoan orchestration, a rate, a tempo, a rhythm, a timbre or an amplitudeassociated with the first state indicated by the communication.

4. The method of paragraph 1, wherein the music that is generated isgenerated in real-time based on feedback from the communication.

5. The method of paragraph 1, wherein the at least one state includes atleast one of a first state associated with a person, a second stateassociated with an environment, a third state associated with a userinterface involving at least one of an interactive touch screen, phone,tablet, digital book, a video game, or a virtual reality or augmentedreality system.

6. The method of paragraph 5, wherein the first state associated withthe person includes at least one of a physical state of a person, amental state of a person, an emotion, a feeling, a biometric, alocation, an activity, a rate of activity, a level of activity or anaction of a user, wherein the second state associated with theenvironment includes at least one of a weather situation, a temperature,an amount of humidity, an amount of light, a time of day, or a time ofyear, wherein the third state associated with the state determined by atext, image, video, video or audio game, or virtual, augmented or mixedreality technology includes at least one of a sixth state associatedwith the state of one or more characters, scenes, or quantifiableattribute.

7. The method of paragraph 1, wherein the communication comprises atleast one of a sensor communication, an Internet of Things (“IoT”)communication, a biometric communication, a voice communication, atextual communication, a photographic communication, or a videocommunication.

8. The method of paragraph 7, wherein the sensor communication isreceived from one or more sensors including at least one of one or moreGPS sensors, one or more distance sensors, one or more motion sensors,one or more movement sensors, one or more speed or velocity sensors, oneor more accelerometer sensors, one or more gyroscope sensors, one ormore biometric/health sensors, one or more facial recognition sensors,one or more cameras, one or more weather sensors, one or moretemperature sensors, one or more ambient light sensors, one or morehumidity sensors, one or more touch sensors, one or more movementsensors, one or more rotation sensors, or one or more microphones oraudio sensors, and wherein the at least one state is determined based onfeedback from the one or more sensors.

9. The method of paragraph 7, wherein the IoT communication is receivedfrom one or more devices comprising at least one of a smart home device,one or more thermometers in one or more rooms, one or more infrared(“IR”) thermometers aimed at one or more positions in the one or morerooms, one or more air flow sensors in the one or more rooms, one ormore air flow sensors in air ducts directed toward the one or morerooms, one or more indoor solar light sensors, one or more outdoor solarlight sensors, one or more outdoor wind sensors, one or moreneighborhood weather station sensors, one or more regional weatherstation sensors, one or more motion detectors detecting presence ofpeople or animals in at least one of the one or more rooms or outsidethe customer premises, one or more humidity sensors in the one or morerooms, one or more smoke detectors detecting smoke in the one or morerooms, one or more gas detection sensors detecting gas in the one ormore rooms, one or more biometric sensors identifying at least oneperson, or one or more health sensors detecting health information forat least one person, and wherein the at least one state is determinedbased on feedback from the one or more devices.

10. The method of paragraph 7, wherein the biometric communication isreceived from at least one of a fitness tracker or anelectroencephalography (“EEG”) device and wherein the computing systemdetermines the at least one state based on feedback from the at leastone of the fitness tracker or the EEG device.

11. The method of paragraph 7, wherein the computing system determinesthe at least one state by at least one of parsing the voicecommunication of at least one person or determining a tone of voice ofthe voice communication of the at least one person.

12. The method of paragraph 7, wherein the computing system determinesthe at least one state by parsing the textual communication or detectingat least one emoji used in the textual communication.

13. The method of paragraph 7, wherein the computing system determinesthe at least one state by determining a displayed state of at least oneperson in the photographic communication.

14. The method of paragraph 7, wherein the computing system determinesthe at least one state by at least one of determining a displayed stateof at least one person in the video communication, analyzing bodylanguage of the at least one person in the video communication, parsingdialogue of the at least one person in the video, or determining a toneof voice of the at least one person in the video.

15. The method of paragraph 1, wherein the communication is furtherindicative of at least one of an age, a sex or location of a person,wherein the one or more first characteristics of a plurality ofcharacteristics of music are further associated with the at least one ofthe age or the sex or the location indicated by the communication, andwherein the music that is generated further has the one or more firstcharacteristics of the plurality of characteristics associated with theat least one of the age or the sex indicated by the communication.

16. The method of paragraph 1, wherein the music that is generatedcontains human-like embellishments.

17. The method of paragraph 16, wherein the human-like embellishmentsare created from at least one of timing jitter, frequency jitter, ortimbral jitter.

18. The method of paragraph 1, further comprising: mapping, with thecomputing system, a plurality of states to a circular pattern, whereinat least one of a position of a state of the plurality of states on acircumference of the circular pattern or an angle of the state on thecircular pattern corresponds to a first subset of characteristics ofmusic associated with the state, and wherein a distance of the statefrom a center of the circular pattern corresponds to a second subset ofcharacteristics of music associated with the state; determining, withthe computing system, the position on the circumference of the circularpattern of at least one state contained within the communication or theangle of the at least one state contained within the communication andthe distance of state contained within the communication from the centerof the circular pattern; and based on the determination of the positionor angle and the distance of the at least one state contained within thecommunication, autonomously generating, with the computing system, musichaving the first subset of particular characteristics and the secondsubset of particular characteristics associated with the determined atleast one state contained within the communication.

19. The method of paragraph 1, further comprising: mapping, with thecomputing system, a plurality of states to a plurality of positions on atwo-dimensional graph, wherein at least one of a first distance of aposition from a first axis corresponds to an input mapped to a firstsubset of characteristics of music, and wherein a second distance of theposition from a second axis corresponds to an input controlling a secondsubset of characteristics of music; determining, with the computingsystem, a state position associated with the determined at least onestate contained within the communication and having a first particulardistance from the first axis and a second particular distance from thesecond axis; based on the determination of the state position having thefirst particular distance from the first axis and a second particulardistance from the second axis, autonomously generating, with thecomputing system, music having the first subset of characteristics andthe second subset of characteristics corresponding to the state positionassociated with the at least one state contained within thecommunication.

In another aspect, a method for generating music might be characterizedby the following sample numbered paragraphs:

20. A method for generating music, said method comprising: generating,with a computing system, a circular pattern having a plurality ofdifferent states mapped to different regions on the circular pattern,wherein at least one of a position of a region on a circumference of thecircular pattern or an angle of the region on the circular patterncorresponds to a first subset of characteristics of a plurality ofcharacteristics of music, and wherein a distance of the region from acenter of the circular pattern corresponds to a second subset ofcharacteristics of the plurality of characteristics of music; analyzing,with a computing system, a communication to determine at least one statecontained within the communication; autonomously determining, with thecomputing system, the position of at least one particular regioncorresponding to the at least one particular state on the circumferenceof the circular pattern or the angle of the region on the circularpattern corresponding to the at least one particular state and thedistance of the at least one particular region corresponding to the atleast one particular emotion from the center of the circular pattern;based on the determination of the position of the determined at leastone particular region on the circumference of the circular pattern orthe angle of the determined at least one particular region and thedistance of the determined at least one particular region from thecenter of the circular pattern, autonomously generating, with thecomputing system, music having the first subset of characteristics andthe second subset of characteristics associated with the at least onestate contained within the communication.

21. The method of paragraph 20, wherein the computing system comprisesat least one of a desktop computer, a laptop computer, a tablet,embedded processing unit or a cellular phone.

22. The method of paragraph 20, wherein the different regions arerepresented by icons, and wherein the icons are at least one of atextual icon representing the at least one emotion or an emojirepresenting the at least one of the emotion.

23. The method of paragraph 20, wherein the first subset ofcharacteristics includes at least one of a pitch, a note envelope, anote pattern, a filter, a harmony, a tone, a density, vocal-likequality, a grouping, an orchestration, a timbre, a rate, a tempo, or anamplitude associated with the first state indicated by the usercommunication.

24. The method of paragraph 20, wherein the second subset ofcharacteristics includes at least one of a pitch, a note pattern, afilter, a harmony, a tone, a density, vocal-like quality, anorchestration, a speed, a timbre, or an amplitude associated with thefirst state indicated by the user communication.

25. The method of paragraph 20, further comprising: autonomouslydetermining, with the computing system, whether at least two states arecontained within the communication; based on a determination that atleast two states are contained within the communication, simultaneouslygenerating, playing, and harmonizing music having the particular firstsubset of characteristics and the particular second set ofcharacteristics associated with each of the at least two statescontained within the communication.

26. The method of paragraph 20, further comprising: autonomouslydetermining, with the computing system, whether at least two states arecontained within the communication; based on a determination that morethan one states is contained within the communication, determining anorder to generate and play music associated with each of the at leasttwo states; transitioning, with the computing system, between generatingand playing music associated with each of the at least two states.

27. The method of paragraph 26, wherein transitioning between playingmusic associated with each of the at least two states contained withinthe communication further comprises: autonomously determining, with thecomputing system, the position or angle of the at least two particularregions corresponding to the at least two particular states on thecircumference of the circular pattern and the distance of two particularregions corresponding to the particular states from the center of thecircular pattern; autonomously determining, with the computing system,regions corresponding to additional states between the at least twoparticular regions corresponding to the at least two particular states;transitioning, with the computing system, between playing musicassociated with each of the at least two states contained within thecommunication by playing music associated with the determined additionalstates between music associated with the at least two particular regionscorresponding to the at least two particular states.

28. The method of paragraph 20, wherein the user communication comprisesat least one of a tactile communication, a sensor communication, anInternet of Things “IoT” communication, a biometric communication, avoice communication, a textual communication, a picture communication,or a video communication.

29. The method of paragraph 28, further comprising: displaying thecircular pattern having the plurality of different states represented byregions to a user on a user interface, wherein the tactile communicationis received from the user selecting at least one regions associated withat least one particular state on the circular pattern displayed on theuser interface.

30. The method of paragraph 28, further comprising: displaying thecircular pattern having the plurality of different states represented byregions to a user on a user interface, wherein the tactile communicationincludes the user selecting at least two regions.

31. The method of paragraph 30, further comprising: tracking, with thecomputing system, the tactile communication of a user; determining, withthe computing system, whether at least one region has been selected;based on a determination that at least two regions have been selected,determining, with the computing system, whether the tactile input, whenselecting the at least two regions, was continuous input; based on adetermination that the tactile input was not continuous, pausing, withthe computing system, the generated music between each user selection ofa region; based on a determination that the tactile input wascontinuous, smoothly transitioning between playing music associated witheach of the at least two states selected by the tactile input.

32. The method of paragraph 31, wherein smoothly transitioning betweenplaying music, further comprises: creating, with the computing system, atime lag between the selection of each region and the generation ofmusic associated with each region. 33. The method of paragraph 32,wherein in the time lag further comprises playing the generated musicfor each selected region for a predetermined amount of time beforetransitioning to a subsequent selected region.

34. The method of paragraph 28, wherein the biometric communication isreceived from at least one of a fitness tracker or anelectroencephalography (“EEG”) device and wherein the computing systemdetermines the first state of the user based on feedback from the atleast one of the fitness tracker or the EEG device.

35. The method of paragraph 28, wherein the computing system determinesthe first state of the user by at least one of parsing the voicecommunication of the user or determining a tone of the voicecommunication of the user.

36. The method of paragraph 28, wherein the computing system determinesthe first state of the user by parsing the textual communication of theuser.

37. The method of paragraph 28, wherein the computing system determinesthe first state of the user by determining a displayed state of a personin the picture communication.

38. The method of paragraph 28, wherein the computing system determinesthe first state of the user by at least one of determining a displayedstate of a person in the video communication, parsing dialogue of theperson in the video, or determining a tone of voice of the person in thevideo.

39. The method of paragraph 20, wherein the user communication isfurther indicative of at least one of a demographic such as age, sex orlocation of a user, wherein the intensity or character of the music isfurther associated with the at least one of the demographics indicatedby the user communication, and wherein the music that is generatedfurther has the intensity associated with the at least one of thedemographics indicated by the user.

40. The method of paragraph 20, wherein the music that is generatedcontains variation and human-like embellishments.

41. The method of paragraph 20, wherein the human-like embellishmentsare created from at least one of timing jitter, frequency jitter, ortimbre jitter, wherein a random signal may influence parameter values.

42. The method of paragraph 20, further comprising: receiving, with thecomputing system, at least one additional user communication from auser, the at least one additional user communication being indicative ofat least one additional state; determining, with the computing system, anumber of user communications over a specified period of time;determining, with the computing system, whether the number of usercommunications exceeds a predetermined threshold; based on adetermination that the number of user communications does not exceed thepredetermined threshold, autonomously determining, with the computingsystem, one or more first characteristics of a plurality ofcharacteristics of music associated with the second state indicated bythe second user communication, and autonomously transitioning fromgenerating music having the one or more first characteristics of theplurality of characteristics associated with the first state indicatedby the user to generating music having the one or more firstcharacteristics of the plurality of characteristics associated with thesecond state indicated by the user; and based on a determination thatthe number of user communications does exceed the predeterminedthreshold, pausing, with the computing system, the music beinggenerated.

In yet another aspect, a method for generating music might becharacterized by the following sample numbered paragraphs:

43. A method for generating music, said method comprising: generating,with a computing system, a two-dimensional graph having a plurality ofdifferent states mapped to different regions on the two-dimensionalgraph, wherein at least one of a first distance of a region from a firstaxis on the two-dimensional graph corresponds to an input mapped to afirst subset of characteristics of a plurality of characteristics ofmusic, and wherein a second distance of the region from a second of thetwo-dimensional graph corresponds to an input mapped to second subset ofcharacteristics of the plurality of characteristics of music; analyzing,with a computing system, a communication to determine at least one statecontained within the communication; autonomously determining, with thecomputing system, a position of at least one particular regioncorresponding to the at least one state contained within thecommunication and having a first particular distance from the first axisand a second particular distance from the second axis; based on thedetermination of the position of the determined at least one particularregion corresponding to the at least one state contained within thecommunication, autonomously generating, with the computing system, musichaving the first subset of characteristics and the second subset ofcharacteristics associated with the position of the at least one statecontained within the communication.

In a further aspect, a method for generating music might becharacterized by the following sample numbered paragraphs:

44. A method for generating new music, the method comprising: defining,with a computing system, a predetermined set of notes; defining, withthe computing system, at least one of a frequency, note pattern, a noteenvelope, a note range, a melody, a harmony, a probability of a noteevent, a tone, an orchestration, a timbre, a rate, a tempo, a rhythmicstructure, a density, a grouping of audio components, a filter, amusical gesture, or an amplitude associated with a particular state;generating, with the computing system, music associated with the atleast one of a note pattern, a note range, a harmony, a tone, aorchestration, a timbre, a speed, a rhythmic structure, a grouping, agesture or the amplitude associated with the particular state; andadding, with the computing system, human-like embellishments to thegenerated music.

In another aspect, an apparatus for generating music might becharacterized by the following sample numbered paragraphs:

45. An apparatus, comprising: at least one processor; and anon-transitory computer readable medium communicatively coupled to theat least one processor, the non-transitory computer readable mediumhaving stored thereon computer software comprising a set of instructionsthat, when executed by the at least one processor, causes the apparatusto: analyze a communication to determine at least one emotion or statecontained within the communication; autonomously determine one or morefirst characteristics of a plurality of characteristics of musicassociated with the determined at least one emotion or state containedwithin the communication; and based on the determination of the one ormore first characteristics of a plurality of characteristics of musicassociated with the at least one emotion or state contained within thecommunication, autonomously generate music having the one or more firstcharacteristics of the plurality of characteristics associated with theat least one determined emotion or state contained within thecommunication.

In an additional aspect, a system for generating music might becharacterized by the following sample numbered paragraphs:

46. A system, comprising: a computing system, comprising: at least onefirst processor; and a first non-transitory computer readable mediumcommunicatively coupled to the at least one first processor, the firstnon-transitory computer readable medium having stored thereon computersoftware comprising a first set of instructions that, when executed bythe at least one first processor causes the computing system to: analyzea communication to determine at least one emotion or state containedwithin the communication; autonomously determine one or more firstcharacteristics of a plurality of characteristics of music associatedwith the determined at least one emotion or state contained within thecommunication; and based on the determination of the one or more firstcharacteristics of a plurality of characteristics of music associatedwith the at least one emotion or state contained within thecommunication, autonomously generate music having the one or more firstcharacteristics of the plurality of characteristics associated with theat least one determined emotion or state contained within thecommunication.

In another aspect, an apparatus for generating music might becharacterized by the following sample numbered paragraphs:

47. An apparatus, comprising: at least one processor; and anon-transitory computer readable medium communicatively coupled to theat least one processor, the non-transitory computer readable mediumhaving stored thereon computer software comprising a set of instructionsthat, when executed by the at least one processor, causes the apparatusto: generate a circular pattern having a plurality of different emotionsor states represented by icons, wherein at least one of a position on acircumference of the circular pattern or an angle of the circularpattern corresponds to a first subset of characteristics of a pluralityof characteristics of music associated with a particular emotion orstates represented by a particular icon, and wherein a distance from acenter of the circular pattern corresponds to a second subset ofcharacteristics of the plurality of characteristics of music associatedwith a particular emotion or states represented by a particular icon;analyze a communication to determine at least one emotion or statecontained within the communication; autonomously determine the positionof at least one particular icon corresponding to the at least oneparticular emotion or state on the circumference of the circular patternand the distance of the at least one particular icon corresponding tothe at least one particular emotion or state from the center of thecircular pattern; and based on the determination of the position of thedetermined at least one particular icon on the circumference of thecircular pattern or the angle of the determined at least one particularicon and the distance of the determined at least one particular iconfrom the center of the circular pattern, autonomously generate musichaving the first subset of characteristics and the second subset ofcharacteristics associated with the at least one emotion or statecontained within the communication.

In an additional aspect, a system for generating music might becharacterized by the following sample numbered paragraphs:

48. A system, comprising: a computing system, comprising: at least onefirst processor; and a first non-transitory computer readable mediumcommunicatively coupled to the at least one first processor, the firstnon-transitory computer readable medium having stored thereon computersoftware comprising a first set of instructions that, when executed bythe at least one first processor causes the computing system to:generate a circular pattern having a plurality of different emotions orstates represented by icons, wherein at least one of a position on acircumference of the circular pattern or an angle of the circularpattern corresponds to a first subset of characteristics of a pluralityof characteristics of music associated with a particular emotion orstates represented by a particular icon, and wherein a distance from acenter of the circular pattern corresponds to a second subset ofcharacteristics of the plurality of characteristics of music associatedwith a particular emotion or states represented by a particular icon;analyze a communication to determine at least one emotion or statecontained within the communication; autonomously determine the positionof at least one particular icon corresponding to the at least oneparticular emotion or state on the circumference of the circular patternand the distance of the at least one particular icon corresponding tothe at least one particular emotion or state from the center of thecircular pattern; and based on the determination of the position of thedetermined at least one particular icon on the circumference of thecircular pattern or the angle of the determined at least one particularicon and the distance of the determined at least one particular iconfrom the center of the circular pattern, autonomously generate musichaving the first subset of characteristics and the second subset ofcharacteristics associated with the at least one emotion or statecontained within the communication.

In another aspect, an apparatus for generating music might becharacterized by the following sample numbered paragraphs:

49. An apparatus, comprising: at least one processor; and anon-transitory computer readable medium communicatively coupled to theat least one processor, the non-transitory computer readable mediumhaving stored thereon computer software comprising a set of instructionsthat, when executed by the at least one processor, causes the apparatusto: define a predetermined set of notes; define at least one of a notepattern, a note range, a harmony, a tone, a pitch, an envelope, aprobability of a note event, an orchestration, a speed, a rate, atimbre, a density, a filter, a contour, a gesture, a density, a groupingof audio components, a group, or an amplitude of an audio featureassociated with a particular emotion or state; generate music associatedwith the at least one of the note pattern, the note range, the harmony,the tone, the orchestration, the speed, a rate, a timbre, a contour, agesture or an amplitude associated with the particular emotion or state;and add human-like variations to the generated music.

In an additional aspect, a system for generating music might becharacterized by the following sample numbered paragraphs:

50. A system, comprising: a computing system, comprising: at least onefirst processor; and a first non-transitory computer readable mediumcommunicatively coupled to the at least one first processor, the firstnon-transitory computer readable medium having stored thereon computersoftware comprising a first set of instructions that, when executed bythe at least one first processor causes the computing system to: definea predetermined set of notes; define at least one of a note pattern, anote range, a harmony, a tone, a pitch, an envelope, a probability of anote event, an orchestration, a speed, a rate, a timbre, a density, afilter, a contour, a gesture, a density, a grouping of audio components,a group, or an amplitude of an audio feature associated with aparticular emotion or state; generate music associated with the at leastone of the note pattern, the note range, the harmony, the tone, theorchestration, the speed, a rate, a timbre, a contour, a gesture or anamplitude associated with the particular emotion or state; and addhuman-like variations to the generated music.

In a further aspect, a method might be characterized by the followingsample numbered paragraphs:

51. A method for generating music, said method comprising: generating,with a computing system, a circular pattern having a plurality ofdifferent emotions or states represented by icons, wherein at least oneof a position of a particular icon on the circular pattern correspondsto a set of characteristics of a plurality of characteristics of musicassociated with a particular emotion or state represented by theparticular icon or position; analyzing, with the computing system, acommunication to determine at least one emotion or state containedwithin the communication; autonomously determining, with the computingsystem, the position of at least one icon or position corresponding tothe determined at least one emotion or state contained within thecommunication on the circular pattern; based on the determination of theposition of the determined at least one particular icon on the circularpattern, autonomously generating, with the computing system, musichaving the set characteristics associated with the determined at leastone emotion or state contained within the communication.

52. The method of paragraph 51, further comprising: autonomouslydetermining, with the computing system, whether at least two emotions orstates are contained within the communication; based on a determinationthat at least two emotions are contained within the communication,simultaneously generating, playing, and harmonizing music having the setof characteristics associated with each of the at least two emotions orstates contained within the communication.

53. The method of paragraph 51, further comprising: autonomouslydetermining, with the computing system, whether at least two emotions orstates are contained within the communication; based on a determinationthat more than one emotion or state is contained within thecommunication, determining an order to generate and play musicassociated with each of the at least two emotions or states;transitioning, with the computing system, between generating and playingmusic associated with each of the at least two emotions or states.

54. The method of paragraph 53, wherein transitioning between playingmusic associated with each of the at least two emotions or states arecontained within the communication further comprises: autonomouslydetermining, with the computing system, the position of the at least twoparticular icons or positions corresponding to the at least twoparticular emotions or states on the circular pattern; autonomouslydetermining, with the computing system, icons corresponding toadditional emotions or states between the at least two particular iconscorresponding to the at least two particular emotions or states;transitioning, with the computing system, between playing musicassociated with each of the at least two emotions or states containedwithin the communication by playing music associated with the determinedadditional emotions or states between music associated with the at leasttwo particular icons corresponding to the at least two particularemotions or states.

In an aspect, a method might be characterized by the following samplenumbered paragraphs:

55. A method for generating music, said method comprising: generating,with a computing system, a pattern having a plurality of differentconcepts represented by icons, wherein at least one of a position of aparticular icon on the pattern corresponds to a set of characteristicsof a plurality of characteristics of music associated with a particularconcept represented by the particular icon; analyzing, with thecomputing system, a communication to determine at least one conceptcontained within the communication; autonomously determining, with thecomputing system, the position of at least one icon corresponding to thedetermined at least one concept contained within the communication onthe pattern; based on the determination of the position of thedetermined at least one particular icon on the pattern, autonomouslygenerating, with the computing system, music having the setcharacteristics associated with the determined at least one conceptcontained within the communication.

56. The method of paragraph 55, wherein the at least on conceptcorresponds to at least one of an emotion, a state of a person, anaction of a person, an attribute of a physical or virtual environment,or interaction in a physical or virtual environment.

57. The method of paragraph 55, wherein the pattern is at least one of acircular pattern, a triangular patter, a rectangular pattern, apentagonal pattern, a hexagonal pattern, an octagonal pattern, atwo-dimensional grid, or a three-dimensional grid.

In an additional aspect, a method for generating music might becharacterized by the following sample numbered paragraphs:

58. A method, comprising: analyzing, with a computing system, acommunication to determine at least one concept contained within thecommunication; autonomously determining, with the computing system, oneor more first characteristics of a plurality of characteristics of musicassociated with the determined at least one concept contained within thecommunication; and based on the determination of the one or more firstcharacteristics of a plurality of characteristics of music associatedwith the at least one concept contained within the communication,autonomously generating, with the computing system, music having the oneor more first characteristics of the plurality of characteristicsassociated with the at least one determined concept contained within thecommunication.

59. The method of paragraph 58, wherein the at least one conceptcorresponds to at least one of an emotion of a person, a state of aperson, an action of a person, an attribute or interaction in a virtualor physical environment.

60. The method of paragraph 58, wherein the concept corresponds to anemotion of a person or a state of a person, and wherein the one or morefirst characteristics of a plurality of characteristics of musicassociated with the determined at least one concept are designed toguide a user from a negative emotion or negative state to a morepositive emotion or positive state.

61. The method of paragraph 58, wherein the music that is generated maybe used as an assistive aid for musical therapy or assistive technology.

In yet another aspect, a method for generating music might becharacterized by the following sample numbered paragraphs:

62. A method, comprising: receiving, with a computing system, at leastone sensor input associated with a user; analyzing, with the computingsystem, the at least one sensor input to determine at least one firststate of the user; determining, with the computing system, a desiredsecond state of the user; determining, with the computing system,whether the at least one first state of the user matches the desiredsecond state of the user; based on a determination that the at least onefirst state of the user does not match the desired second state of theuser, continuously controlling, with the computing system, thegeneration of music to guide the user to the desired second state basedon the at least one sensor input.

63. The method of paragraph 62, further comprising: based on adetermination that the at least one first state of the user does matchesthe desired second state of the user, continuously controlling, with thecomputing system, the generation of music to have one or morecharacteristics associated with the desired second state.

64. The method of paragraph 62, wherein continuously controlling thegeneration of music further comprises both fixed evolving aspects andvariable changes driven by at least at least one sensor input.

65. The method of paragraph 62, wherein the at least one sensorcomprises at least one of a biometric sensor, an electrode, a GPSsensor, a distance sensor, a motion sensor, a movement sensor, a speedor velocity sensor, an accelerometer, a gyroscope, a facial recognitionsensor, or a video or still image.

66. The method of paragraph 62, wherein the desired second state of theuser corresponds to a desired stress level or amount of arousal of theuser, and wherein continuously controlling the generation of the musiccauses the generated music to guide the user from a first stressfulstate to a second more relaxed state based on the at least one sensorinput.

67. The method of paragraph 62, wherein the desired second statecorresponds to state of meditation, and wherein continuously controllingthe generation of the music causes the generated music to guide the userfrom a first non-meditative state to a second meditative state based onthe at least one sensor input.

68. The method of paragraph 62, wherein the desired second statecorresponds to state of focus, and wherein continuously controlling thegeneration of the music causes the generated music to guide the userfrom a first unfocused state to a second focused state based on the atleast one sensor input.

69. The method of paragraph 62, wherein the desired second statecorresponds to an emotion, and wherein continuously controlling thegeneration of the music causes the generated music to guide the userfrom a first negative emotion to a second positive emotion.

70. The method of paragraph 62, wherein the desired second statecorresponds to a desired number of steps to take per minute, and whereincontinuously controlling the generation of the music causes thegenerated music to guide the user to take the desired number of stepsper minute.

71. The method of paragraph 62, wherein continuously controlling thegeneration of the music causes the generated music to sync to at leastone of a running speed of the user, a heartbeat of the user, a breathingpattern of the user, or a brainwave pattern of the user.

72. The method of paragraph 62, wherein the desired second statecorresponds to at least one of a slower breathing pattern, a slowerheartbeat, a faster breathing pattern, or a faster heartrate, andwherein continuously controlling the generation of the music causes thegenerated music to guide the user to at least one of a slower breathingpattern, a slower heartbeat, a faster breathing pattern, or a fasterheartrate.

73. The method of paragraph 62, wherein the desired second statecorresponds to at least one of a decreasing intensity of a workout or anincreasing intensity of a workout, and wherein continuously controllingthe generation of the music causes the generated music to guide the userthrough at least one of the decreasing intensity of a workout or theincreasing intensity of a workout.

74. The method of paragraph 62, wherein the desired second statecorresponds to at least one of a location or a position of the user, andwherein continuously controlling the generation of the music causes thegenerated music to guide the user from a first location to a seconddesired location.

75. The method of paragraph 62, wherein the desired second statecorresponds to at least one goal-oriented state, and whereincontinuously controlled music allows the sound generated to guide theuser to towards the desired goal state.

76. The method of paragraph 75, wherein the desired goal to bemaintained corresponds to at least one of maintaining a heart rate,maintaining a breathing pattern, maintaining a running pace, maintaininga step pace, maintaining a rate of periodic activity, or maintaining avelocity or amount of an activity.

77. The method of paragraph 62, wherein the desired second statecorresponds to at least one of a posture of the user or a body positionof the user, and wherein continuously controlling the generation of themusic causes the generated music to guide the user to at least one of animproved posture or improved body position or improved gesture movement.

78. The method of paragraph 62, wherein the at least one sensor input isat least one biometric input, and wherein the at least one biometricinput is received from at least one of a biometric input readercomprising at least one of a blood pressure monitor, a heart ratemonitor, EKG sensor, PPG sensor, a fitness tracker, CO2 monitor, pulseoximeter, muscle sensor, temperature sensor, respiration sensor, camera,accelerometer, gyroscope or an electroencephalography (“EEG”) device andwherein the computing system determines the at least one state of aperson based on feedback from the at least one of the biometric input.

79. The method of paragraph 62, wherein the one or more characteristicsof music include at least one of a frequency, a note pattern, a noteenvelope, a filter, a harmony, a tone, vocal-like quality, surpriseelement, randomization, consistency, crescendo, decrescendo anorchestration, a speed, a rate, a rhythm, or an amplitude associatedwith the at least first biometric input.

In yet a further aspect, an apparatus for generating music might becharacterized by the following sample numbered paragraphs:

80. An apparatus comprising: at least one processor; and anon-transitory computer readable medium communicatively coupled to theat least one processor, the non-transitory computer readable mediumhaving stored thereon computer software comprising a set of instructionsthat, when executed by the at least one processor, causes the apparatusto: receive at least one sensor input associated with a user; analyzethe at least one sensor input to determine at least one first state ofthe user; determine a desired second state of the user; determinewhether the at least one first state of the user matches the desiredsecond state of the user; based on a determination that the at least onefirst state of the user does not match the desired second state of theuser, continuously control the generation of music to guide the user tothe desired second state based on the at least at least one sensorinput.

In an additional aspect, a system for generating music might becharacterized by the following sample numbered paragraphs:

81. A system, comprising: a computing system, comprising: at least onefirst processor; and a first non-transitory computer readable mediumcommunicatively coupled to the at least one first processor, the firstnon-transitory computer readable medium having stored thereon computersoftware comprising a first set of instructions that, when executed bythe at least one first processor causes the computing system to: receiveat least one sensor input associated with a user; analyze the at leastone sensor input to determine at least one first state of the user;determine a desired second state of the user; determine whether the atleast one first state of the user matches the desired second state ofthe user; based on a determination that the at least one first state ofthe user does not match the desired second state of the user,continuously control the generation of music to guide the user to thedesired second state based on the at least at least one sensor input.

In another aspect, a method for generating music might be characterizedby the following sample numbered paragraphs:

82. A method, comprising: receiving, with a computing system, a firstbiometric input associated with a user; analyzing, with the computingsystem, the first biometric input to determine a first state of theuser; autonomously determining, with the computing system, one or morefirst characteristics of music associated with the determined firststate of the user; based on the determination of the one or more firstcharacteristics of music associated with the determined first state ofthe user, autonomously generating, with the computing system, musichaving the one or more first characteristics associated with the firststate of the user; receiving, with the computing system, at least onesecond biometric input associated with the user; analyzing, with thecomputing system, the at least one second biometric input to determineat least one second state of the user; determining, with the computingsystem, whether the second state of the user is different from the firststate of the user; based on a determination that the second state of theuser is different from the first state of the user, autonomouslydetermining, with the computing system, one or more secondcharacteristics of music associated with the determined at least onesecond state of the user; based on a determination of the one or moresecond characteristics of music associated with the second state of theuser, autonomously transitioning between generating music having the oneor more first characteristics of music associated with the first stateof the user to generating music having the one or more secondcharacteristics of the plurality of characteristics associated with thesecond state of the user.

In an additional aspect, a method for generating music might becharacterized by the following sample numbered paragraphs:

83. A method, comprising: continuously receiving, with a computingsystem, one or more biometric inputs associated with a user; analyzing,with the computing system, the one or more biometric inputs to determineat least one first state of the user; determining, with the computingsystem, a desired second state of the user; determining, with thecomputing system, whether the at least one first state of the usermatches the desired second state of the user; based on a determinationthat the at least one first state of the user does not match the desiredsecond state of the user, continuously generating music having one ormore evolving characteristics of music associated with the at least onebiometric input, wherein the one or more evolving characteristics ofmusic guide the user to the desired second state based on the at leastat least one biometric input.

In an aspect, a method for generating music might be characterized bythe following sample numbered paragraphs:

84. A method, comprising: analyzing, with a computing system, a sensorinput or a communication to determine at least one state; autonomouslydetermining, with the computing system, one or more firstcharacteristics of a plurality of characteristics of music associatedwith the determined at least one state; and based on the determinationof the one or more first characteristics of a plurality ofcharacteristics of music associated with the at least one state,autonomously generating, with the computing system, music having the oneor more first characteristics of the plurality of characteristicsassociated with the at least one determined state.

85. The method of paragraph 84, wherein the music that is generated isgenerated in real-time based on feedback from the sensor input or thecommunication.

86. The method of paragraph 84, wherein the one or more firstcharacteristics of the plurality of characteristics of music reflect anintensity of the determined at least one state.

87. The method of paragraph 84, wherein the one or more firstcharacteristics of the plurality of characteristics of music reflect avalence of the determined at least one state.

While certain features and aspects have been described with respect toexemplary embodiments, one skilled in the art will recognize thatnumerous modifications are possible. For example, the methods andprocesses described herein may be implemented using hardware components,software components, and/or any combination thereof. Further, whilevarious methods and processes described herein may be described withrespect to particular structural and/or functional components for easeof description, methods provided by various embodiments are not limitedto any particular structural and/or functional architecture but insteadcan be implemented on any suitable hardware, firmware and/or softwareconfiguration. Similarly, while certain functionality is ascribed tocertain system components, unless the context dictates otherwise, thisfunctionality can be distributed among various other system componentsin accordance with the several embodiments.

Moreover, while the procedures of the methods and processes describedherein are described in a particular order for ease of description,unless the context dictates otherwise, various procedures may bereordered, added, and/or omitted in accordance with various embodiments.Moreover, the procedures described with respect to one method or processmay be incorporated within other described methods or processes;likewise, system components described according to a particularstructural architecture and/or with respect to one system may beorganized in alternative structural architectures and/or incorporatedwithin other described systems. Hence, while various embodiments aredescribed with—or without—certain features for ease of description andto illustrate exemplary aspects of those embodiments, the variouscomponents and/or features described herein with respect to a particularembodiment can be substituted, added and/or subtracted from among otherdescribed embodiments, unless the context dictates otherwise.Consequently, although several exemplary embodiments are described aboveand elsewhere herein, it will be appreciated that the invention isintended to cover all modifications and equivalents within the scope ofthe appended claims.

Additional illustrative embodiments will now be described herein withreference to exemplary breathing entrainment sonification systems andassociated computers and other processing devices, as shown in FIGS. 13through 24. It is to be appreciated that the embodiments describedbelow, like others described herein, are presented by way of exampleonly, and should not be construed as limiting in any way. Other types ofentrainment sonification systems can be implemented using the disclosedtechniques. It should also be noted that terms such as “music” and“musical” as used herein are intended to be broadly construed, so as toencompass a wide variety of different sound arrangements.

Although illustrative embodiments are described primarily in the contextof breathing entrainment sonification, it will be readily apparent thatthe disclosed techniques can be adapted for use in entrainmentsonification contexts involving a wide variety of other mindfulnessrelated activities.

A breathing entrainment sonification system in some embodimentsdisclosed herein may be implemented in the form of a musicalcommunication system configured to support a breathing entrainmentapplication. For example, breathing entrainment sonification systemsdisclosed herein may be implemented in the form of musical communicationsystems of the type described above, but particularly configured forbreathing entrainment sonification.

In some embodiments, a breathing entrainment sonification systemcomprises a musical communication system particularly configured for anapplication involving guided breathing exercises.

A breathing entrainment sonification system in some embodiments isconfigured in accordance with a breathing entrainment model. The systemconfigured in accordance with the breathing entrainment model moreparticularly comprises a closed-loop system featuring two differenttypes of sound cues, namely, a sound cue of a first type to direct theuser's breathing pattern (an “entrainment component”) and one or moresound cues of a second type to provide feedback to the user on theircurrent status during the exercise (one or more “auxiliary components”).

The two different types of sound cues are designed to overcome many ofthe challenges in biofeedback systems including negative feedback loopsand ambiguity in the function and meaning of the sonification.Additional or alternative sound cues for entrainment and/or auxiliarycomponents may be used in other embodiments.

The entrainment component illustratively relates to a “leader” role ofsound as described elsewhere herein, functioning to guide, instructand/or direct the user's activity. With the exception of a few edgecases including sleep onset detection and external interrupts forreal-time ramp generation, the user's performance during the activitygenerally does not influence this component.

An auxiliary component's function illustratively relates to a “follower”role as described elsewhere herein, responding to and reflecting theuser's activity. It is influenced by the user's activity during theexercise, providing feedback on how the user is doing (progress) andwhat they should do to improve (adjustment).

Other embodiments can be configured to utilize only an entrainmentcomponent. The auxiliary components can therefore be eliminated in someembodiments. Additionally or alternatively, some embodiments can beimplemented using an open-loop arrangement rather than a closed-looparrangement. Accordingly, some embodiments disclosed herein areconfigured to utilize only entrainment components, and such embodimentscan be implemented using closed-loop or open-loop arrangements.

Many conventional auditory solutions attempt to assist withactivities—such as meditation, relaxation and exercise—by providing aconducive soundtrack, but none offer intuitive feedback through soundthat guides the user towards their goal. Illustrative embodiments of thebreathing entrainment sonification systems disclosed herein address thisneed by offering users the ability to increase bodily awareness throughreal-time auditory response to sensor input. Such breathing entrainmentsonification systems are flexible enough to sonify any datastreamthrough synthesis techniques, without using bulky audio clips.

Mindfulness, the practice of being aware of the present moment throughmental focus, is an example of an activity that can benefit fromsonifying sensor input, in this case taking a breath. Sound cues inillustrative embodiments can help users follow a programmed breathingexercise (entrainment, guide or leader signal), understand how well theyare doing at the activity (response, feedback or follower signal), andexperience heightened awareness of the breath, thus encouraging a stateof mindfulness.

By presenting users with access to breath-by-breath guide and auditoryfeedback, illustrative embodiments overcome the drawbacks ofconventional use of a soundtrack conducive to the activity. Thebreathing entrainment sonification techniques disclosed herein assistusers in working toward a goal.

For example, sound cues as disclosed herein are intuitive from the firstlisten, and associations are strengthened through practice. The temporaldimension of sound affords both a means to guide transformation overtime, as well as to induce regularity through rhythmic and patternrepetition. Research has shown music's power in regulating respiration,improving gait control, and other therapeutic and restorative goals.

Illustrative embodiments utilize a generative sound model for guidedbreathing that harnesses sound's ability to calm the mind, and focus thelistener on their breath.

The breathing entrainment model described is an assistive applicationconsisting of multiple sound cues which simultaneously guide the user'sbreathing patterns and reflect how they are doing during the exercise.

FIG. 13 shows a breathing entrainment sonification system 1300 thatutilizes an example arrangement of sound cues configured to provideentrainment, progress and adjustment in an illustrative embodiment. Inthis embodiment, there are two types of auditory cues heard by the user:a primary cue 1301 for entrainment (e.g., leader, guide) and multipleauxiliary cues 1302 (e.g., follower, feedback). The primary cue 1301 andauxiliary cues 1302 in some embodiments are combined together in asignal combiner with the resulting combined signal driving an audiodevice 1304 for generation of sound for audible presentation to a user1305. Other types of signal processing arrangements are possible. Theaudio device 1304 illustratively comprises at least one speaker, and maybe part of a larger processing device that implements at least a portionof the breathing entrainment sonification system, such as a mobiletelephone, tablet computer or other type of computer.

The primary cue 1301 is an example of what is more generally referred toherein as a first sound cue of a first type, and illustrativelycomprises a primary entrainment cue for the breathing entrainmentsonification system 1300. The primary breathing entrainment cueillustratively comprises a particular entrainment signal configured todirect a breathing pattern of the user 1305 towards a desired breathingpattern. For example, the particular entrainment signal may comprise aselected one of a plurality of distinct entrainment signals availablewithin the breathing entrainment sonification system 1300. Additionallyor alternatively, one or more characteristics of the particularentrainment signal are illustratively adjustable by the user 1305, suchas via a touchscreen or other user interface of a mobile telephone ortablet computer that implements at least a portion of the breathingentrainment sonification system 1300.

The auxiliary cues 1302 are examples of what are more generally referredto herein as additional sound cues of a second type, with each suchadditional sound cue illustratively comprising an auxiliary breathingentrainment cue for the breathing entrainment sonification system 1300.A given auxiliary breathing entrainment cue is illustratively configuredto provide an indication to the user 1305 of his or her current statusrelative to a designated goal and/or guidance of the user 1305 towardthe designated goal.

In other embodiments, the auxiliary cues 1302 may be eliminated, and thesystem 1300 can operate to provide entrainment sonification using onlythe primary cue 1301.

In some implementations of the breathing entrainment sonification system1300 of FIG. 13, the primary cue 1301 and the auxiliary cues 1302 areprovided via the audio device 1304 to the user 1305, and one or morefeedback signals are received from one or more sensors of the breathingentrainment sonification system 1300. The system 1300 can then adjustone or more characteristics of one or both of the auxiliary cues 1302,and additionally or alternatively one or more characteristics of theprimary cue 1301, based at least in part on the one or more feedbacksignals received from the one or more sensors. Such sensors are notexplicitly shown in the embodiment as illustrated in FIG. 13, but anexample of such a sensor is shown in the embodiment of FIG. 14 to bedescribed below. Open-loop implementations of system 1300 and othersystems herein are possible in other embodiments.

FIG. 14 shows a breathing entrainment sonification system 1400 thatimplements a breathing entrainment bio-feedback loop in an illustrativeembodiment. This diagram depicts an overall breathing entrainmentsonification system configuration based on a breathing entrainmentmodel, such as, for example, a valence/intensity model in which aprimary entrainment cue is mapped to an intensity parameter, and anauxiliary entrainment cue is mapped to a valence parameter. It is to beappreciated, however, that numerous other types of entrainmentsonification models can be used in other embodiments.

The breathing entrainment sonification system 1400 comprises asonification processor 1402 coupled to an audio device 1404 thatincludes at least one speaker. The sonification processor 1402 isconfigured to analyze feedback signals received from one or more sensors1406, each illustratively referred to as a “peripheral,” and possiblyfrom one or more other peripherals of other types. The sonificationprocessor 1402 includes an analysis stage 1410 for generatingentrainment and auxiliary components of the type previously described inconjunction with FIG. 13. Such components are provided to a synthesisengine 1412 that generates one or more signals for driving the audiodevice 1404.

In the system 1400, sound is generated by the audio device 1404 and auser 1405 listens to the sound and interprets the primary and auxiliarysound cues contained therein. The user 1405 responds to the sound cueswhich provide instruction and feedback to the user 1405 on how he or sheshould perform an activity. The activity of the user 1405 is thenmeasured by the one or more sensors 1406 and possibly other peripheralswhich provide feedback to the sonification processor 1402 as illustratedin the diagram. The analysis stage 1410 measures activity changes andprovides further processing to generate mapping signals for the soundcomponents, including entrainment and auxiliary components. The soundcomponents are fed in as inputs into the synthesis engine 1412 wherethey are mapped to sonic gestures or other types of sound cues.

A wide variety of different sensors and/or other peripherals can be usedin illustrative embodiments, examples of which are provided elsewhereherein. Also, the term “sensor” as used herein is intended to be broadlyconstrued, and accordingly can encompass a wide variety of sensingdevices of different types and configurations. As indicated above, anopen-loop implementation of system 1400 is possible, in which the one ormore sensors 1406 may be eliminated. Some implementations of system 1400in other embodiments can eliminate the one or more auxiliary componentsand utilize only an entrainment component.

FIG. 15 shows examples of entrainment signal shapes utilized inbreathing entrainment systems such as systems 1300 and 1400 inillustrative embodiments. These are various different types of signalswhich can be used to guide breathing air flow in and out. Such examplesinclude entrainment signals having rising portions corresponding torespective inhale phases of a desired breathing pattern and fallingportions corresponding to respective exhale phases of the desiredbreathing pattern.

A given such entrainment signal of FIG. 15 can comprise a triangularoscillator signal as shown in example (a) or a sinusoidal oscillatorsignal as shown in example (b). A linear ramp signal with pause phasesbetween adjacent instances of the inhale and exhale phases can be used,as shown in example (c). Another type of linear ramp signal that can beused as an entrainment signal in illustrative embodiments comprises oneor more variable air flow speed changes as shown in example (d). Othertypes of entrainment signals include a complex ramp signal with variedor otherwise variable trajectories as shown in example (e), and acomplex ramp signal with one or more external interrupts as shown inexample (f). Combinations of these and other entrainment signals canalso be used, and the particular examples of FIG. 15 should therefore beconsidered as non-limiting illustrations only.

FIG. 16 shows an implementation example of a breathing entrainmentsonification system 1600 utilizing a breathing entrainment model in anillustrative embodiment. In this embodiment, system 1600 comprises anentrainment component 1601, illustratively providing a “leader/guide”function, and an auxiliary component 1602, illustratively based at leastin part on valance of a valence/intensity model as described elsewhereherein, and providing a “follower/feedback” function. The system 1600further comprises an audio device 1604 that includes at least onespeaker.

The diagram in FIG. 16 illustrates the signal flow of the breathingentrainment model for this particular implementation example. Theentrainment component 1601 is applied via a filtered noise component1610 to a first input of a signal combiner 1612. The auxiliary component1602 is coupled to a mapping component 1620, illustratively denoted“StagMap” in the figure, which drives a bank of oscillators 1622 thathave their respective outputs coupled to respective corresponding inputsof a filter 1624. The filter 1624 also receives as its other inputs theentrainment component 1601 and the auxiliary component 1602. An outputof the filter 1624 is applied to a second input of the signal combiner1612, which generates a composite signal for driving the audio device1604. The mapping component 1620 provides a staggered mapping functionthat illustratively includes at least two alternative mappings 1621A and1621B, also referred to as Alternative 1 and Alternative 2,respectively. Each of these alternatives provides multiple differentmappings of valence to signal amplitude based at least in part on amountof feedback as illustrated.

Additional details regarding the operation of system 1600 can be foundin FIGS. 18 and 19, which show the entrainment component and auxiliarycomponents mapped to valence/intensity emotion model parameters.

In some embodiments, the different sound cues are constructed by mappingvarious combinations of sound design techniques to parameters controlledby the entrainment and auxiliary components. Examples of sound designtechniques utilized to convey information include timbral density,tonality vs. nosiness, timbre heterogeneity, spectrum morphing, eventdensity, formant filtering and synthesis, high/low pitch, spectrumdistribution, rate, tempo, frequency, relative phase, filter sweeps,note length, note distance, randomness, articulation, loudness, pitchheight, number of voices, pitch distribution, and spatializationmappings, although numerous other sound characteristics can be used.

The entrainment component and the auxiliary components of illustrativeembodiments will now be described in greater detail.

Entrainment Component

An entrainment component (e.g., leader) illustratively comprises aperiodic control signal intended to guide breathing activity. Theperiodic signal illustratively outlines the shape of a breath, and ismapped to continuous parameters in the sound model.

The entrainment component drives a strongly implied gesture in the soundmodel, outlining the trajectory of an inhale, exhale and pause timesin-between breaths. This is achieved by mapping the oscillating leadersignal to musical parameters which carve out a trajectory a user canreadily perceive in the overall sound.

Musical parameters associated with up and down movement may be relatedto in and out movement.

The entrainment component in some embodiments is parametric and may becontrolled in real-time. These parameters may be set at the beginning ofexercise or be dynamically controlled throughout the exercise.

The entrainment component can be configured to adapt to a variety ofbreathing exercises which are outlined by different signal shapes. Anupward slope signals an inhale, a downward slope signals an exhale,while no change in slope relates to a pause or hold time in-betweenbreaths.

Once a breath shape signal for a given exercise has been determined, thewaveform may be repeated to create a continuously changing signal.

Different entrainment signal slopes cue different speeds of airflow. Seethe examples illustrated in FIG. 15, which were introduced above.

These entrainment signal shapes each generally include a repeated signalused to synchronize breathing.

For example, a low frequency oscillator entrainment signal can be basedon a table-lookup algorithm with some form of interpolation (e.g.,linear, cubic, etc.) The entrainment component may take the shape ofcommon oscillators including triangle and sinusoidal waveforms, asillustrated in respective examples (a) and (b) of FIG. 15. Amplitude caninfluence how dramatic the breathing gesture cue is. The “dramatic-ness”of the entrainment cue may be reduced if the user does not need tofollow the breathing cue anymore to achieve a desired outcome. Theamplitude of the entrainment signal being gradually reduced after sleeponset is detected. The frequency of the oscillator can be adjustedduring the session to correspond with the intended breathing exercise,e.g., the frequency may gradually decrease over time if the breathingexercise is designed to slow down the user's breathing.

Other examples include ramping entrainment signals, illustrativelycomprised of calculated line segments, typically with breakpoints inunits of time or as a ratio relative to the other segments of a breathcycle. Ramp signals may be generated in real-time by interrupts,messages received by an external protocol with information to shape theinhale and exhale ramps. They can include a wait portion, where thesignal waits for input from an external interrupt before it continueswith the next line segment or group of segments. This behavior can beused in situations when breathing is correlated with muscle contraction(e.g., exercise equipment). Inhale (ramp up), exhale, (ramp down), andhold times (pauses) may be specified by providing the ratio values foreach of the parts of the breath cycle or specifying time for eachsection of the breathing cycle before beginning exercise. Rampingentrainment signals can include linear ramps, which have linear linesegments between points, as in example (c) of FIG. 15. More complexramps include ramps with varied trajectories, e.g. multiple differentslopes (or speeds) for inhaling and/or exhaling (fast to slow air flow),as illustrated in examples (d), (e) and (f) of FIG. 15. These and othercomplex ramps can be configured utilizing hand drawn curves, exponentialcurves, Bezier curves, or a wide variety of other curve types.

Although only a single entrainment component is referred to above, it ispossible in some embodiments to use multiple such components.

Auxiliary Components

An auxiliary component (e.g., follower) illustratively comprises asecond continuous sound cue influenced by an external input, functioningto provide feedback to the user about their progress or current status.For example, an auxiliary component can be used to provide an indicatorof the degree which the user is doing the breathing exercise correctly,thereby conveying progress and/or adjustment information. Progressand/or adjustment can be based at least in part on physiological statedetermined from a biosensor, or can be time-based if there is nobiosensor input available. As mentioned previously, numerous other typesof sensors can used, including, for example, RF sensors.

An auxiliary component does not overpower or cloud the entrainmentcomponent in the sound model but modifies the overall sound so that theuser is aware that something has changed.

FIG. 17 illustrates examples of two major ways in which the user caninterpret the information conveyed by a continuous auxiliary component:progress and adjustment. See (1) vs (2a) and (2b) in FIG. 17.

There can be multiple auxiliary components in a single model. Forexample, two separate auxiliary components are utilized in someembodiments, such as the illustrative embodiment of FIG. 13.

Different types of progress can be conveyed, as illustrated in FIG. 17.For example, auxiliary components may be perceived as unidirectional orbidirectional to provide information to the user about progress andadjustment instruction. FIG. 17 shows different ways in which thelistener can perceive the feedback sound cues. Amount in (1) can berelated to progress, while directional cues in (2a) and (2b) can berelated to adjustment.

With regard to progress, the general magnitude of an auxiliary componentcan answer questions relating to progress of the user. Such questionsare implicitly posed by the user as he or she participates in anactivity.

For example, one question of interest to a given user is “How am Idoing?” The magnitude of the sound cue can depict if a user is doinggood or bad. Noticeably different regions on the progress meter cansignal if the user is close to or far away from the goal. The user canhear from the progress meter if their activity is being done correctlyor close to correctly.

Another example of a question of interest to a user is “How far away amI from my target goal?” The continuous auxiliary component offers moreresolution on quantitative information. The continuous control signalallows for users to hear the distance they are away from the goal toprovide awareness the sense of how far away they are from the goal oroptimal state. Duration can be set as a target goal to convey to theuser how much time is left in the activity.

With regard to adjustment, the position on a multi-state auxiliarycomponent can provide instructional information to the user about howthey should adjust their activity, as illustrated in FIG. 17.

An example of a question in this area is “How should I adjust mybreathing?” Interpolated different regions controlled by continuousauxiliary component can provide feedback cues to indicate whetherbreathing is faster or slower than it should be. A bi-directionalfeedback component can signal to the user that they need to go faster orslower (and how much faster or slower). The directional cues portion ofFIG. 17 shows an example of bi-directional sound cues where the goallies somewhere in the middle (˜0.5) so a user can tell if he or she isare breathing too fast (<0.5) or too slow (>0.5). The position of goalon a slider from 0-1 may not be exactly at 0.5, and there may be a biasto the left or right. Another example shown in FIG. 20 illustrates acustom goal positions diagram. This diagram illustrates other situationswhere the position of the goal state on a feedback meter may becustomized to some other location between the minimum and maximumvalues.

As noted above, various types of external inputs can be applied to theauxiliary components. Examples of external inputs into the auxiliarycomponents include data streams that encapsulate information about thecurrent status or progress of the user during the exercise. Inputsrelate to the intended outcome or goal of the breathing exercise, or anyother mindfulness related activity. There may be different breathingexercises for different situations which require different combinationsof peripherals to convey progress. Different intended outcomes (goals)from the breathing exercise may include synchronization (entrainment),relaxation, energizing, focus, or meditation, arousal or simply doingthe activity for a given amount of time. A wide variety of differentinputs can be used, including by way of example various measurableparameters that relate to intended outcome of the breathing activity.

Various metrics can additionally or alternatively be used, including byway of example one or more of:

1. Relaxation, stress and sleep onset and stage (derived from one ormore of the following: EEG, GSR, EMG, PPG, EKG, other heart sensor).

2. Entrainment amount as reflected in breathing rate (derived from oneor more of the following: EKG, conductive or resistive respiration belt,accelerometer, gyroscope, RF sensor, camera, pressure sensor); walkingrate (derived from same sensors above); and other movement rate(s).

3. Time (timers, passage of time, etc.) if goals are time-based.

4. Focus and meditation amount (via EEG).

5. Movement amount.

6. Increased heart rate variability (HRV).

7. Lower blood pressure.

8. Reduced muscle tension.

9. Any other metric which relates to an intended outcome of thebreathing exercise.

Again, the foregoing and other listings herein are examples only, andother arrangements can be used.

Although multiple auxiliary components are referred to above, it ispossible in some embodiments to use only a single such component.

FIGS. 18 and 19 illustrate an implementation example based on mappingusing a valence/intensity emotion model as described elsewhere herein. Atwo-component breathing entrainment sonification system in anillustrative embodiment can use such a valence/intensity emotion model.In such an embodiment, the entrainment component can be mapped to globalintensity parameter, while the auxiliary or feedback component can bemapped to valence. The diagrams of FIGS. 18 and 19 illustrate the signalflow for this two-component breathing entrainment sonification systemusing the valence/intensity emotion model. Other types of models can beused in other embodiments.

Referring initially to FIG. 18, a breathing entrainment sonificationsystem 1800 in this embodiment comprises an entrainment component 1801and an auxiliary component 1802 based at least in part on valence usingthe above-noted valence/intensity emotion model. The model asillustrated by the graph in the upper right portion of the figure isconfigure to map the entrainment component 1801 to an intensityparameter, and to map the auxiliary component 1802 to a valenceparameter.

In the system 1800 utilizing the valence/intensity emotion model, anintensity level 1803 is driven by an entrainment signal of theentrainment component 1801, with the result being input to a sound model1807. A sensor 1806 provides input to a feedback computation module 1809that is configured to perform an error calculation for a current state,and also to drive auxiliary component 1802 to generate feedback asindicated. Such feedback is also provided to the sound model 1807, whichis utilized to adjust one or more characteristics of at least one of theentrainment component 1801 and the auxiliary component 1802.

As indicated previously, entrainment sonification systems such as system1600 and system 1800 described above can alternatively be configured inother embodiments to utilize only entrainment components, and/or toutilize an open-loop arrangement rather than a closed-loop arrangement.

Turning now to FIG. 19, an example of the feedback computation isillustrated. The upper portion of the figure shows the goal as a desiredbreathing pattern characterized by a rate of entrainment signaloscillator. Actual breathing rate as determined from sensing dataprovided by sensor 1806, and an amount of feedback is determined as thedistance between a sensor measurement and a horizontal dashed linecorresponding to the goal. The absolute value (“Abs”) of the rescaleddistance conveys an indication to the user as to how far off from thegoal that user is based on the sensor measurement. The rescaled distanceitself can be positive or negative relative to the goal and indicates tothe user a direction in which the user should move with its breathing(e.g., faster or slower) order to get closer to the goal.

FIG. 20 illustrates custom goal positions that may be used in someembodiments. This diagram shows how the position of the goal state on afeedback meter may be varied. For situations where the goal is not atthe maximum, minimum or midpoint of a feedback slider, the goal regionmay be customized to be an arbitrary point between the maximum andminimum. The horizontal goal arrow in this embodiment may therefore beimplemented as a slider or other control mechanism in a user interfaceof a mobile telephone, tablet computer or other type of computer thatimplements at least a portion of the breathing entrainment sonificationsystem.

FIG. 21 shows an example of state interpolation that may be used inillustrative embodiments. This diagram shows density distributionmappings which illustratively correspond to an example implementation ofa mapping component, such as the StagMap of FIG. 16. Different statesare associated with regions having values which always add up to 1. Itis possible to interpolate between different states to create differentregions with noticeably different characteristics.

FIG. 22 shows customized interface examples in illustrative embodimentsin which the interface is on a mobile phone, although a wide variety ofother types of user devices with different interfaces can be utilized inother embodiments. The baseline entrainment sound and feedbackcomponents' sonic attributes can be customized to fit different musicalpreferences. FIG. 22 more particularly comprises four distinctscreenshots, two in FIG. 22A and two in FIG. 22B. Screenshot #1 of FIG.22A shows a portion of an interface allowing a user to choose a pace,and screenshot #2 of FIG. 22A shows a portion of an interface allowing auser to adjust focus in terms of duration. Screenshots #3 and #4 of FIG.22B show portions of an interface allowing a user to adjust focus interms of musical mood and sensor input, respectively.

Numerous alternative arrangements are possible. For example, simplecustomization may involve selecting different presets which modify themood of the overall sound heard (see #3 in FIG. 22B).

It is also possible to morph preset customization. For example,customization may involve gesture-based control over default entrainmentsound (without influence by a follower component).

There are a wide variety of different ways in which one may customizethe sound. For example, some embodiments utilize a single model withdifferent presets that can be mixed by interpolating values ofparameters defining presets. A single model can load in many presetsthat alter musical parameters different from those being controlled bythe leader and follower components. In such an arrangement, users canselect and mix up to a particular number (e.g., 4) different presets ofthe same model. This can be achieved by loading different presets fromthe same model into a two-dimensional interface where they can exploredifferent combinations of the presets by moving the X/Y coordinateposition on a touch screen.

FIG. 23 includes a screenshot #3.1 illustrating an example interface toallow for mixing different combinations of 4 presets where each cornerin a customization view can load a different preset. Once the desiredmix is achieved, the user can save the configuration using the portionof the interface shown in screenshot #3.2.

It is also possible to utilize multiple models. For example, differentmodels can be used that all feature two high-level parameters. In suchan arrangement, each preset has two parameters which user can modify onX/Y.

FIG. 24 shows an example of a two-dimensional interface where a user cancontrol a feedback amount to preview sound, so as to allow the user tocustomize the baseline and reward sounds separately. On the left side ofthe figure, a feedback position associated with a slider relative to anon/off icon can be controlled by a user in order to allow the user tohear various types of transitions in certain sound cues. The right sideof the figure illustrates an arrangement in which a position of a sliderbetween two circles controls customization parameters for respectivedefault sound and feedback sound. One of the two circles in thisarrangement denotes a default position with a feedback value of 0 (i.e.,no feedback) and the other circle denotes a position with a feedbackvalue of 1 (i.e., maximum feedback). Other customized feedback valuesare achievable by adjusting a position of the slider between the twocircles.

As mentioned previously, these and other particular features ofillustrative embodiments are presented by way of example only, andshould not be viewed as limiting in any way. For example, although someembodiments are described in the context of breathing entrainmentsonification, such embodiments can be adapted in a straightforwardmanner for use in a wide variety of other entrainment sonificationcontexts. Also, although some embodiments herein are in the form ofclosed-loop entrainment sonification systems, other embodiments can beimplemented as open-loop entrainment sonification systems which do notrequire feedback from sensors or other peripherals.

Additional illustrative embodiments will now be described herein withreference to FIGS. 25 through 29. These figures show aspects of musicalcommunication systems and associated applications including breathingentrainment sonification.

Illustrative embodiments provide methods, apparatus, systems andcomputer program products for musical communication and associatedapplications. Such embodiments include musical communication systemsthat are configured to support at least one of a plurality of distinctapplications, including, for example, self-regulation of biometrics andbiofeedback training applications, with information conveyed throughabstract sound providing assistance in relaxation, focus, meditation,fitness and posture; physical therapy and gesture training applications,with sound providing continuous feedback to performing a gesture withminimal error; entrainment applications in which sound is used tosynchronize activity to a rate; and navigation applications, with soundproviding an awareness of position and directionality of movement in amulti-dimensional space.

These embodiments can be illustratively configured to provide a userwith an awareness of a current state and a guide towards an optimal ortarget state, illustratively through the use of appropriate feedback toadjust user behavior. These and other embodiments can additionally oralternatively provide rewards for achieving and/or maintaining theoptimal or target state, and can provide additional or alternativefunctionality.

Many other musical communication systems and associated applications aresupported in other embodiments, including, without limitation,screenless interfaces for augmented reality systems, and environmentalmonitoring systems.

FIG. 25 shows an example of a musical communication system 2500 in oneembodiment. The musical communication system 2500 in this embodimentcomprises a data stream 2501 (“A”), a logic engine 2502 (“B”), an audioengine 2503 (“C”) and a human 2504. The human in this system is anexample of what is more generally referred to herein as a “user,” andaccordingly will be referred to as user 2504 below. The user 2504 isassumed to be associated with a processing device, such as a mobiletelephone, a tablet computer or another type of computer, including anaudio device comprising at least one speaker. Feedback 2505 based atleast in part on activity of the user 2504 becomes part of the datastream 2501. The various components of the FIG. 25 system are describedin more detail below.

The data stream 2501 is illustratively defined as including a signalcomprising at least one numerical value that updates over a reasonablyregular period of time. The signal is assumed to be received inreal-time. For example, the data may be currently streaming from areal-time data source. Multiple signals of different types can beincluded in the data stream 2501, such as different signals fromdifferent ones of a plurality of sources, such as an exercise machine,physical therapy equipment, a wearable sensor, a camera, a motionsensor, an environment sensor, and interface, and the cloud. A givensignal of the data stream 2501 can also include combinations of signalsfrom these and additional or alternative data sources. Also, there maybe multiple distinct instances of data stream 2501 in some embodiments,such as different data streams from respective different ones of theabove-noted sources.

The logic engine 2502 performs stream processing. For example, it takesin the data stream 2501 or multiple such data streams and outputsstreams of data mappable to parameters in a sound synthesisimprovisation model. It can perform functions such as signal filtering,scaling, statistics generation and analysis, and long term analysis.Output signals are mostly continuous in nature, but can also includediscrete events sent to the audio engine 2503.

The audio engine 2503 implements at least one procedural sound modelinvolving sound parameter mappings. For example, built-in structures areillustratively improvised on, and streams of improvisations are shapedby inputs coming into the model. Algorithms implemented by the audioengine 2503 are illustratively configured to produce sound cues thatlisteners can take meaning from. The audio engine 2503 is illustrativelyconfigured to provide smooth or seamless transitions between soundstates (e.g., from a user experience point of view). For example, audiooutput sounds can update in sub-milliseconds in some embodiments becausethe sounds are being synthesized rather than sampled, thereby providinga better user experience that is customizable on a nuanced level.

The ears and brain of user 2504 represents a human audience component ofthe system 2500. In some embodiments, the user 2504 can have activecontrol over the sound through an interface, but such control is anoptional feature. Sounds from the audio engine 2503 produce cues audibleto the human ear. The user 2504 interprets these cues and is therebyinformed about the current state. For example, state could be used todescribe the state of the processed data stream and/or the state of themusical communication system. State in some embodiments could be definedas the state of the data stream in the context of what it is trying toconvey, although numerous other state definitions can be used in otherembodiments.

Additional details regarding the FIG. 25 embodiment and otherillustrative embodiments will be described below.

FIG. 26 illustrates example interactions between the logic engine 2502,the audio engine 2503 and the user 2504 within the musical communicationsystem 2500. The audio engine 2503 in some embodiments is implemented inthe form of a computer system that synthesizes electronic sound in aprocedural way. For example, instead of reading from a pre-determinedmusical score, a computer generates and improvises new sound and musicbased on an underlying set of inputs, rules and constraints. Theprocedural music used in this on context is engineered to bring anintuitive understanding to what the data stream 2501 is conveying.

Streams of music/sound are illustratively modulated in real-time by theinput data processed via the logic engine 2502. For example, the logicengine 2502 in this embodiment is configured to generate one or moreparameter signals based at least in part on a current state relative toa goal state. A given parameter signal can indicate a distance betweenthe current state and the goal state, or other similar parameter,possibly scaled to fall within a predetermined parameter range such asthe 0 . . . 1 range illustrated for one or more example parameters inthe figure. The corresponding parameter signals in this embodiment areprovided by the logic engine 2502 to the audio engine 2503, whichillustratively implements discrete changes and continuous changes,possibly based at least in part on respective improvisation and slowtimbre evolutions as indicated in the figure.

Such changes are made audibly apparent to the user 2504 via an audiodevice comprising at least one speaker, and the resulting abstract soundconveys various types of information to the user 2504, examples of whichare shown in the figure. The user 2504 in response to such sound cuesadjusts his or her activity in a manner that attempts to drive anupdated current state closer to the goal state for that activity.

In some embodiments, the system 2500 is configured to differentiatelarge gestures consisting of more obvious or noticeable sound cues andmusical events, illustratively comprising examples of what are moregenerally referred to herein as “communications,” from smaller, lessnoticeable changes, illustratively referred to herein as“embellishments,” which employ more subtle changes to the sound, andwhich may involve randomness and jitter used to generate variation inthe sound and music. Gesture illustratively refers to a more macro levelcontrol in which cues are designed to have clear communicativefunctions, which could provide a sound guide to follow, such as, forexample, “sync with” or “entrain to,” or provide feedback to convey thecurrent state of the user or to prompt the user to make an adjustment.The amount of change in such an arrangement may be encoded into one ormore corresponding sound cues.

It should be noted that a wide variety of different types of feedbackcan be provided in illustrative embodiments herein. For example, in someembodiments, feedback in may be driven at least in part by a timer. Thetimer in some embodiments of this type can be configured by incrementinga slider value mapped to continuous gesture and/or discrete sound cues.Similar to notifications when certain goals are met, the timer maytrigger discrete musical events at various subdivisions. As a moreparticular example, illustrative embodiments can implement musicaltimers that could be applied to technology such as a microwave or fooddelivery service when there is a countdown featured. The music can serveto provide a sense of how much time is left in a given task. This timerconcept may also be employed in numerous other applications, including,for example, the running, breathing and masking applications describedin conjunction with FIG. 29 below.

The logic engine represents the data pre-processing which is done tomassage the data into a simple format so that the audio engine can takein as a high-level parameter.

Improvisational aspects relate to variation in the musical notes orscore. These features include statistical models, randomness, grammars,etc. that affect the discrete note events in the audio engine. Theimprovisational components may be used as communication to provideinformation to the listener or embellishments to promote long termlistenability, for example, allowing for the sound/music to play for along time without ever repeating. Probabilities may be changeddynamically to increase or decrease the likelihood of a note event tooccur. These features can be thought of as embellishments or ornamentswithin predefined structures if they do not inhibit the maincommunication sound cues so drastically that it clouds the meaning orsense of directionality of the overarching gesture. The structure ofmeaning embedded in the sounds remain even with the generativecomponents.

Slow evolutions similarly provide constant variation but on a continuoustime frame. These slow evolutions can be employed as embellishments ifthe musical parameters they are mapped to subtly influence the musicalsystem, e.g. changes to them are barely noticeable by the listener.Gradual changes in timbre can occur over time so that the texture isconstantly evolving. These can be achieved by having slowly changingcontrol signals influence different continuous parameters. Randomnessand jitter signals which change on as faster time frame may besimilarity be applied to different parameters to insure that there isnot a static timbre. Like the improvisational features which createvariation on the note level, the amount of influence of these signalscan be minimized so that it does not overpower the overall gesturecontrolled by changes in the input data stream in order to maintain thecommunication.

Sound gesture refers to a type of continuous sound cue where the overallperceived change of the sound provides a sense of position anddirectionality. This illustratively comprises changes of musical eventshappening in time as well as other features, such as timbre, which arechanging continuously. Typically, a single gesture is defined by acontrol signal which is then mapped to multiple musical parameters. Thecombined effect of all the musical features moving together, changing asa result from changes in the data stream, create the sense of a gesture.Gesture can provide an awareness of state movement directionality.

The musical communication system in some embodiments shapes sound in away that can be broken down into two distinct time scales or components,namely, discrete time scales or components and continuous time scales orcomponents, also referred to as respective discrete and continuouslayers.

The discrete layer refers to the musical abstraction of a “note,” thefoundational building block in music theory. This layer concerns itselfwith things like chords, musical scales, and counterpoint. For example,the discrete layer illustratively denotes “what is played,” comprisesmusical symbolic encoding of a score at the note level, and conveyslarge changes.

The continuous layer can be largely interpreted as a (perceptually)continuous signal underneath the discrete time layer. Continuous timeparameters of the continuous layer map to features like timbralbrightness of an instrument playing a note. The continuous layeroutlines the gesture of a particular discrete phrase, and can be used tosignal a change in state faster than the discrete layer. For example,the continuous layer illustratively denotes “how it is played,”comprises fine tune control over sound attributes, and conveys small,nuanced state changes.

Features are controlled in both discrete and continuous time to conveystates.

States refer to what that the music or sound generated is intended toconvey. There is usually an optimal goal state and the sound generatedis intended to convey the users current state with respect to the goalstate.

States may be single or multi-dimensional. Logic may be applied to datastreams to get it into the right format to be inputted into the musicalcommunication system. A single input may determine the goal, oralternatively a combination of multiple inputs may determine single ormultiple goals.

Examples of states that may be utilized in illustrative embodimentsinclude mental state (e.g., emotional state, relaxation level, focuslevel, stress level); physical state (e.g., activity rate, posture,gesture, amount (amplitude) of movement or change in state, velocity ofmovement or change in state, angle (gyroscope/accelerometer), GPSlocation, elevation); virtual position (e.g., position in interface,such as angle/rotation of movement (with respect to previous location),screen area location or location of game element, three or moredimensional location in virtual space); and level meter (e.g.,quantifiable data stream, level of something computed, environmentcondition such as temperature, humidity, light, etc.) Sound featuresthat convey information in some embodiments may be similar to featurespresent in the emotion models described elsewhere herein.

For example, an emotion model can have two high level input parameters:valence and intensity. Most emotions can be depicted with these twoparameters: Happy=high valence & high intensity, Sad=low valence, lowintensity, Angry=low valence, high intensity, Calm=high valence, lowintensity, etc.

Valence can be positive and negative, and related to a goal state.

Intensity is similar to energy.

Both valence and intensity can provide an awareness of current state andmovement between states.

Particular examples of sound features include timbre e.g. brightness,harshness, spectral density, articulation; rate/density e.g. tempo, noteonset events, number of instrument voices; pitch, e.g. melody,harmonization; and randomization/jitter.

This is only one way in which musical mappings can be constructed toconvey meaning. Numerous other models can be used in other embodimentsfor conveying meaning in contexts outside of emotion that are focused ongoal-based activities.

Some embodiments make use of leader and/or follower roles. In sucharrangements, sound generated in a musical communication system asdisclosed herein can play different functions in goal-oriented tasks.

In the case of a follower, the sound symbolically reflects user state.Sound is intended to mirror a given state, acting like a directtranslation of the data into sound and music. Sound provides what isreferred to herein as “informative entertainment,” where abstractcommunication of information regarding current state(s) is conveyed tothe listener(s) through an entertaining medium. The follower role ismainly for less directed and free-form settings. A goal or optimal state(if featured) likely indirectly relates to input state and may likelyinvolve additional logic algorithms. If applicable, sound may rewarduser for maintaining goal over time.

In the case of a leader, sound is intended to guide the user towards anoptimal state. Sound informs the user how close the current state is tothe goal state, and informs the user about a change in state. Soundrewards the user for maintaining goal state over time. The goal/optimalstate are more likely to directly relate to input state.

In some embodiments, musical communication technique are classified intothe following two categories which are based off of perception ofauditory cues: absolute sounds and relative sounds.

Absolute sounds are distinct sound characteristics we can immediatelyhear and classify. Such sounds facilitate categorical association(instinctive or easily learned intuitive association). For example,sound at given position can provide information about current statewithout the need for movement, and no reference sounds are needed.Examples include recognizable harmonic, melodic and rhythmic motifs,etc. Absolute sounds may consist of short bursts of sounds withcharacteristics that symbolize the event or action that has occurred.

Relative sounds are distinct audible changes in sound from a movingstate. Movement between states provides information about current stateand direction of movement. Relative changes may reflect changes in statewith respect to previous state, and/or changes in state with respect togoal state. Relative sounds provide responsiveness and fluid evolutionsto a sound driven by a continuous input data stream. Relative soundsprovide a sense of gesture that would be more difficult to convey withan absolute sound. Examples include amplitude, pitch, filter cut-off,tempo, rate, articulation, etc. Unlike an absolute sound, a relativesound cue requires some sort of reference to quantify amount of change.

Additional details regarding exemplary applications that may besupported by a musical communication system in illustrative embodimentswill now be described. Aspects of these applications involveconfiguration of the logic engine and/or audio engine. These particularapplications and their respective features and functionality arepresented as examples and should not be viewed as limiting in any way.

In a valence/intensity application of the type described elsewhereherein, a two input parameter model is designed to convey positivity ornegativity of a state and the intensity of the state. For example, in agraphical representation in which an x-axis intersects with a y-axis atan origin point, an x-axis can be used to represent valence, withpositive values to the right of the origin point and negative values tothe left of the origin point, and a y-axis can be used to representintensity, with higher intensity values above the origin point and lowerintensity values below the origin point. More detailed examples of twoparameter emotion models and associated mappings are described elsewhereherein.

In some embodiments of this type, intensity (y-axis) controls pitchheight, rate of note onsets, tempo, timbre or brightness. Valence(x-axis) controls amount of randomness of pitch, note envelope orarticulation, harshness of timbre, detuning amount, amplitude of certainvoices, e.g., bass.

Another example relates to guided entrainment for periodic activities,for a single user. In this application, the system mechanics areconfigured such that the music/sound/stimulus features a rhythmicelement that plays at optimal “goal” rate. The musical communicationsystem detects what rate the activity is actually being performed at andcompares it to a goal rate. Sound attributes (e.g., apart from rate)provide cues to the listener about their current rate with respect tothe goal.

The function of sound in this embodiment is to guide a single user toperform cyclic activity at an optimal rate (e.g., cycles per minute).

The goal state is illustratively an optimal rate that may be influencedby the activity performed.

For example, the optimal state may be predetermined and not influenceddirectly by activity being performed. Leader examples in this caseinclude goal determined by fixed state or position, predefined curveoutlines changing goal state over time, goal determined by user profile,e.g. sensor calibration or demographic information inputted.

Alternatively, the optimal state may be influenced in real-time byactivity performed. A leader/follower example in this case includes onein which the rate of music/sound/stimulus speeds up or slows down withuser's activity. The goal is to maintain any rate consistently for acertain amount of time with minimal fluctuation.

FIG. 27 shows a number of different breathing entrainment examplescollectively denoted as breathing entrainment modules 2700. Threedifferent modules are shown, each of which includes a gesture generator(“gesturizer”) 2702 coupled to a synthesizer 2704, with the input to thegesturizer 2702 in each module being a “breath rate” signal thatillustratively corresponds to a desired breathing pattern.

The first module at the top of the figure is a general implementationcomprising a gesturizer 2702-1 coupled to a synthesizer 2704-1. A signalgenerated by the gesturizer 2702-1 outlines a breathing gesture andcontrols the synthesizer 2704-1 to guide and/or reflect breathing (soundcould play both leader and follower roles). The pattern of inhale andexhale at the input breath rate is mirrored by the gesturizer 2702-1which produces a variable signal to control the synthesizer 2704-1.Multiple synthesizers may be used in place of a single synthesizer inother embodiments. The gesturizer 2702-1 takes in the breath rate signaland produces a control signal. The control signal generated by thegesturizer 2702-1 may be an oscillator signal, illustratively having asinusoidal, triangular or custom shape that reflects the ratio ofinhales, exhale and hold time in-between breaths, various examples ofwhich were previously described in conjunction with FIG. 15. The ratiobetween inhales and exhale times may change depending on rate and can belooked up in a table or determined using other techniques.

Each of the other modules in FIG. 27 generally operates in a mannersimilar to that described above for the first module. However, thesecond module is more particularly configured with a gesturizer 2702-2that include a low frequency oscillator (LFO), and a synthesizer 2704-2that includes a noise source and a filter, with the filter being drivenby an output of the LFO of the gesturizer 2702-2, in order to generatean output of the type described above.

In the third module at the bottom of FIG. 27, a gesturizer 2702-3includes a clock signal source driven by the breath rate signal asshown. The resulting clock signal is fed into a switch which alternatesbetween two parallel envelope signals ENV 1 and ENV 2 that are appliedto respective inputs of a synthesizer 2704-3. The two envelope signalsare more particularly applied to respective first inputs of twodifferent mixers of the synthesizer 2704-3. Those mixers also receive attheir respective second inputs two different oscillator signals fromrespective oscillators denoted OSC 1 and OSC 2. The outputs of themixers are combined in a signal combiner to provide the output of thesynthesizer 2704-3.

Signals from the gesturizers 2702 in the various example breathingentrainment modules described above may be used to modulate amplitude,timbral spectrum, filter parameters and other sound attributes of thesynthesizers 2704.

Another example embodiment provides a running companion, in which musicis intended to guide running rate (e.g., steps per minute) whilemotivating the user to maintain goal rate. The goal rate may bedetermined by one or more the following: rate(s) may be set by user atstart of work out, as a single steady rate or a fluctuating ratedetermined by a predefined curve. Predefined trajectories of differentwork out types, e.g. steady rise and fall, rapid increases and decreasesetc. can be used.

FIG. 28 shows a number of example goal curves, each indicating changesin optimal state over time. The example goal curves illustrated in thisfigure include a warm-up/cool-down curve, an interval training curve, asteady page curve, an oscillating build curve, and a free-form drawingcurve, with these curves being denoted in the figure as respectiveexample 1 through 5. Accordingly, illustrative embodiments can beconfigured to utilize goal curves determined by preset workout curveswith variable time lengths which can be set by the user (examples 1-4).It is also possible to use free-form drawing of an ideal trajectory overtime in interface (example 5). Rate(s) may be dynamically adjustedduring work out. External logic involving additional input can includeheart-rate. For example, optimal heart rate during work out may bedetermined by sensor calibration, age/demographics and workout time. Theaim is keep heart rate in a given range by speeding up and slowing downrunning. If the detected heart rate is too high at a given point intime, the music signals the user to slow down and vice versa. Thirdparty direct influence, e.g., from trainer or competitor, can also beused.

The generated sounds in this embodiment are intended to convey, forexample, how fast you are going (raw steps per minute or speedcategories), the need to speed up, slow down (signed deviation fromtarget running speed), rewards, motivation (that you are doing a goodjob for extended period of time), and completing running milestones(e.g., hitting thresholds for specific running targets).

A given musical communication system in an embodiment of this typeprovides entrainment functionality, illustratively through rhythmic orcyclic main pulse/beat of music/sound/stimuli at optimal rate e.g.overall tempo. It can utilize directional cues such as absolute sound togive the user a sense of position, and relative sound to instruct theuser with features other than tempo, to move faster or slower to reachoptimal rate (see 2b. in Gesture Mapping diagram of FIG. 17). Inaddition, sounds rewards can be provided to the user for performingactivity successfully at optimal rate. The particular amount of rewardsound/effect may be a separate continuous parameter (See 1. Level Meterexample in Gesture Mapping diagram of FIG. 17.)

The goal state may also simply be reflected by the middle state of abinary parameter.

Synthesis techniques to convey meaning may involve various combinationsof mapping density, pitch, rate/tempo, frequency, tonality, formants,pitch, randomness, articulation, loudness, phase, timbre morphing, voicemultiplicity (number of instrument voices), and spatializationparameters.

Another implementation example utilizes movement rate cues and isreferred to herein as “RunFast.” In this embodiment, sound and musicconveys a message to “speed up” or “slow down” through sonic cues otherthan tempo. A combination of discrete and continuous controls oversounds cue the listener to speed up, slow down or maintain a rate ofmovement.

In one possible instrument example of this type, three main rhythmicprofiles with three distinctly different densities act as readilyapparent cues about current state. The rhythmic profiles have built-ingenerative aspects to create variation. Such an embodiment can utilizebi-directional timbre with two contrasting states (with high and lowpitch profiles) of a designated lead instrument, thereby allowing forcontinuous morphing of instrument voice spectrum depending on positionof input state.

In other embodiments, pitch, timbre and density of notes may be mappedto single input parameter, or drum pitch, density and envelope can becontinuously controlled by an input parameter. Additional instrumentexamples can be configured using variations on the above examples.

An example instrument signal flow for such an implementation of RunFastcan illustratively include multiple distinct “steppers” comprisingrespective signal generators that control respective instruments (e.g.,bass lead, gated pad, hi-hat, claps, kick, etc.) Each of the signalgenerators is illustratively driven by a common clock signal, generatedby a tempo component, and receives other control signals from a directorcomponent. Instrument outputs are illustratively combined in a signalcombiner, and then subject to a limiting operation in a limitercomponent.

A breathing entrainment gesture may sonify the airflow speed or volumeof a breath. Both sonifications may also be employed simultaneously,where the airflow speed cues the user on how fast to inhale or exhale,while the volume displays imagery of how full the lungs are of air. Asecond gesture, independent of the entrainment cue, may provide feedbackto the user on their activities performed. The second gesture may beviewed as a type of auxiliary component as disclosed herein.

Similar to the breathing entraining application, a custom movementtrajectory can be mapped to a musical gesture to guide body form duringstrength training exercises involving movement and contractions back andforth. A custom curve outlining the oscillating trajectory of movementis mapped to a musical gesture which the user is intended to follow orentrain to. This musical gesture can outline the trajectory of movementor muscle contractions. Feedback can be provided if the activity isdetected to let the user know how close he or she is to the goaltrajectory, and instruct the user to make adjustments. Exercises wherethis application may be used include, for example, weight lifting,squats, pushups, presses, or activities where form is critical such ascircuit training exercise on machines such as pull-downs.

In some embodiments, functionality for customizing entrainment signalsis provided. For example, individuals can adjust the shape of theentrainment signal mapped onto the musical gesture manually orautomatically. If manual, the system can be adjusted through a touchinterface. If a breath or movement detection system is available totrack different sections on the breath or movement cycle, this signalmay be used to automatically adjust the length or ratio of each sectionto match that of the user. This may be used to calibrate the system tofit each user optimally. Numerous other customization arrangements arepossible in these and other embodiments disclosed herein.

Yet another application relates to guided entrainment for multipleusers. The goal in this application is to guide multiple people to moveto an optimal rate together and convey the degree to which the multiplepeople are in sync. Like the single user example, the optimal rate maybe determined and/or influenced by activity measured. If the optimalrate is set by activity measured, it may reflect the activity of one ormore users. For example, the optimal rate may be set by a single user'sactivity who assumes a leader role, and the other users must follow therate set by the single leader. Alternatively, the optimal rate may bedetermined by multiple users' activity, possibly based on collectedstatistics. It is also possible that the optimal rate may not beinfluenced by user activity and is instead determined by a predefinedvalue or a trajectory over time.

A more particular example is a “Breathing Together” application in whichsound is utilized to guide multiple people to breathe together at anoptimal rate. The goal/optimal rate may be set by predefined curves ortrajectories (sound plays leader role similar to running example), ormay be influenced by the actual breathing rates of the participants. Forexample, if the majority of participants are breathing much faster thanthe goal rate, the rate may speed up (rate feature takes follower role).A group reward may be provided. For example, as more people breathe atthe correct rate, the sound will be more rewarding, e.g. the soundbecomes more positive and grows in intensity.

In some group entrainment systems disclosed herein, an instructor maycontrol the breathing entrainment sonification system in different ways.For example, an instructor may control the parameters of a predeterminedguiding entrainment signal through a touch screen interface. Throughsuch an interface, an instructor may control the speed and shape of theentrainment signal driving the sound cue which the listeners must moveand/or breathe to. The instructor may also control the feedbackcomponent which may influences the amount of energy in the soundgenerated. Feedback can also be used to reinforce the instructor if itis driven by the instructor's movements. For example, an instructor maywear sensors detecting movement, such as speed and position, measured bya camera, gyroscope, or accelerometer, and the amount of movementdetected would be mapped to the feedback component.

Group activity detected may additionally or alternatively influence thefeedback component of the sound. For example, data from individual userscan be mapped to particular instruments that make up the sonification,however, the number should be limited (e.g., to about 3) in order toensure that the sound cue does not get lost in the mix. For largergroups, data from individual users may be averaged or otherwise combinedtogether and then used to drive the entrainment or feedback components.

Examples of user interface (UI) control panels for illustrativeembodiments of running, breathing and masking sonification systems aredescribed in more detail below in conjunction with FIG. 29.

Other applications include level meters and monitors, and illustrativelyinvolve state monitoring. Sound conveys the currentstate/position/status of the user or environment. Continuously mappablesound with optimal state (e.g., value somewhere between −1 and 1 or 0and 1) is used.

As a more particular example, an application is configured to monitorrelaxation level as determined by biometric sensor(s) (e.g., EEG, EMG,ECG, PPG, GSR, HRV, breathing rate). The monitored relaxation level is asingle parameter that is mapped to potentially many features in sound toconvey relaxation level and directionality of change in state.

In one implementation referred to herein as “SonicPull,” relaxationlevel is determined from relative alpha level to other frequency bandsin EEG signal and scaled between 0-1. The relaxation level is fed into amodel as the main input and is mapped to various sound attributes toconvey level of relaxation to the user. The continuous directionalnature of sounds pull the user towards maximum relaxation state.Directionality may be conveyed through pitch, with dissonance beingresolved into consonance as input level rises. Directionally canadditionally or alternatively be conveyed utilizing a filter cutoff on anoise oscillator. Sound becomes denser as more voices fade in at stackedlevels relating to input. Such arrangements illustratively useprobabilities to control density. For example, discrete time control canbe used in a monophonic arrangement, with probabilities (e.g., P=0.1,P=0.5, P=0.9) set by parameter and applied to a rhythmic motif.Additionally or alternatively, layering can be used to control density.For example, continuous time control can be used in a polyphonicarrangement, in which different regions on a slider or other type ofscaled parameter map to different voices. Thus, in such an arrangement,a main input parameter can control the amplitude levels of respectiveones of a plurality of voices.

One example SonicPull implementation includes four main components(e.g., pads, noise, plucks, arps) with sub-mappings controlled by globalrelaxation level. A “pad” may comprise a sustained chord or tonegenerated by a synthesizer, typically used for background. A “pluck” inelectronic music illustratively refers to a percussive and melodiccategory of sound that is widely used. An “arp” or arpeggiator isillustratively a feature, available on some hardware synthesizers andsoftware instruments, allowing conversion of a held note or chord intoan arpeggio. Other components may be used in other embodiments.

Outputs of the four main components are combined in a first signalcombiner. An output of the first signal combiner is applied to a reverbcomponent, while another output of the signal combiner bypasses thereverb component and is applied to a second signal combiner that alsoreceives an output of the reverb component. An output of the secondsignal combiner represents the output of this example SonicPullimplementation.

Another example involves sonification of risk level, illustratively inthe form of a screenless risk monitoring tool for traders who need to beaware of overall risk level while making quick decisions. An optimalrange is determined somewhere between maximum threshold and minimumvalue. (See 2b. in Gesture Mapping diagram of FIG. 17). Different zonesconvey different amounts of risk, e.g. at low levels far away fromthreshold, sound encourages upward movement, while sound is mostrewarding and positive in mid-range safe zone, and sound gets moreintense with growing negative cues at high levels past safe zonethreshold.

Yet another example involves a navigation application in which sounds isused to guide a person in a space to reach the goal position.

In one implementation referred to herein as “SoniNav,” sound conveys howclose a current position is to a goal position while providing feedbackon direction and speed of movement. The user listens to two separatesound cues that are changed continuously by position and movement datato be guided towards the goal position. There is illustratively aprevious position, a current position and a goal position, with anabsolute distance between the current position and the goal position,and an angle between a first line connecting the previous position andthe current position, and a second line connecting the previous positionand the goal position.

For example, a first cue denoted Cue 1 is the absolute distance from thecurrent position to the goal position, and is one parameter controllingthe rate, pitch and envelope of clicks. When the current position iscloser to the goal position, intensity of the clicks grows and they arefaster, higher in pitch and more articulated. A second cue denoted Cue 2is the angle or direction of movement with respect to the goal position,and controls a second aspect of the sound which includes abi-directional tone with two contrasting pitch and timbre profiles. Atone end (0 degrees), the sound signifies movement in the correctdirection, while at the other end (180 degrees), sound conveys movementin the wrong direction. Interpolation between the two contrastingprofiles allows for continuous morphing between the two states to givean awareness of location between states.

The angle between the goal state and the current position with respectto the previous position gets rescaled to determine mix or positionbetween of two contrasting profiles. If the angle is closer to 0degrees, the sound signifies the correct direction. If the angle iscloser to 180 degrees, the sound cue signifying movement in the wrongdirection is heard more. Velocity controls loudness of Cue 2, so that isonly heard when there is movement.

Another example of a level meter application is an application thatguides a user to perform a gesture with minimal error. In this example,continuous sound cues convey whether the gesture is being performedcorrectly, in real-time as the gesture is being performed by the user.The farther away the movement performed is from the optimal gesturetrajectory, the less rewarding the sound generated will be. Similarimplementations to those previously described in conjunction with thenavigation and running applications may also be applied in thisapplication for shaping sound in continuous and discrete time.

The above-described embodiments and their features and functionality areexamples only.

Other embodiments can include alternative implementations of the logicengine and/or the audio engine.

Some embodiments can comprise arrangements such as discrete densityimplementations and/or pitch height control. Both arrangements areillustratively driven by a clock (“CLK”) in these embodiments. Thedensity implementation utilizes a probabilistic trigger element(“may-trig”) that controls an instrument (e.g., a hi-hat). The pitchheight is also probabilistically-controlled in this embodiment.

In other embodiments, discrete improvisation may be implemented in amusical communication system, to provide selection between notegroupings, chords, scales, melodies or counterpoints based onprobabilities.

Additionally or alternatively, a cross-fading rhythm approach may beutilized in some embodiments. This approach involves linearlyinterpolating between rhythms, using a probabilistic model with acontinuous probability value. The assumption is that the rhythms can berepresented by binary values on a fixed time scale. This cross-fadingrhythm approach is in contrast to tree-like probabilistic approachesthat are used to generate rhythmic/melodic content that does notnecessarily share the same time scale.

Referring now to FIG. 29, examples of UI control panels are shown forrunning, breathing and masking sonification systems in illustrativeembodiments. The running, breathing and masking sonification systemscomprise respective Run, Breathe and Mask applications running on asmart phone as illustrated in the figure. The upper and lower parts ofthe left side of FIG. 29 show different UI screens of the Runapplication. Similarly, the upper and lower parts of the middle andright side of the figure show different UI screens of the respectiveBreathe and Mask applications.

The Run smart phone application UI allows users to generate a musicaltrainer, which can help pace exercise and keep track of progress throughaudio feedback. Music builds if activity detected matches a target goal.This example shows the target goal to be a running cadence (steps perminute rate) set by a pre-determined curve. If heart rate detection isavailable, a different mode allows for heart rate goal trajectory to beset. Music tempo fluctuates with heart rate to help the user match thetarget heart rate. A logic engine is used to determine the differencebetween actual heart rate and target heart rate, and to adjust the tempoaccordingly. Feedback may be driven by difference in heart rate or stepsper minute depending on configuration. Other metrics may be used to setgoals including distance and intensity.

The Breathe smart phone application is designed to assist with stressregulation and train focus. The smart phone UI illustrated in thisexample allows the user to specify breathing rate trajectory over time.In Breathe, compatible sensors used to drive feedback in someimplementations can include sensors for respiration, HRV, GSR andrelaxation level from EEG, movement, etc. Additional examples aredescribed elsewhere herein.

The Mask smart phone application is designed to block out unwanteddistractions. The smart phone UI illustrated in this example allows theuser to create and sculpt a masking soundscape to aid with sleep andfocus. The user may tune adaptive noise to fit his or her environmentand listening preferences. If biometric sensors are available, feedbackmay be provided by modulating the sound to convey information to theuser in a non-obtrusive manner through gradual transformations. Feedbackcan also be employed to help with mind wandering, or to gently wake youup from slumber. Sensors which may be used as plug-ins include EEG oreye tracker sensors. Masking and personalization features characterizingMask may be used as a standalone application or be combined with theBreathing application and/or one or more other applications.

These and other illustrative embodiments disclosed herein can includetools for augmented reality which use generative music for abstractcommunication. For example, some embodiments are in the form of musicsoftware configured to intuitively convey information through sound andmusic from data streams in interactive platforms.

Such embodiments can be configured to interact with wearables, virtualassistants, smart cars and IoT devices, as well as numerous other datasources, in order to leverage available data to improve lives. Audioengines in illustrative embodiments are configured to generate“flexible” music in the form of sound that can be shaped in real-time,so as to provide users with awareness of themselves and theirenvironments.

Illustrative embodiments enhance user experience by providing meaningfulcommunication, in some cases through interactive AI producing engagingsound, state-of-the-art algorithmic models and real-time sonic attributemapping.

Some embodiments provide highly personalized listening, includinggenerative audio with infinite variation, dynamic sound and music shapedwhile playing, and intuitive control through simple interfaces.

These and other embodiments are illustratively implemented in a highlyflexible manner, possibly utilizing a software development kit (SDK)that is highly portable and cross-platform, and provides easy mapping tosensor inputs and efficient DSP algorithms.

Some embodiments can be configured to avoid the need for sample loops orpre-rendered musical material. Examples of embodiments of this typeinclude musical communication systems that generate material usingprobabilities to continuously create variation. In such an arrangement,the musical communication system is illustratively synthesis-based usingsignal processing algorithms to shape the sound in real-time.

Examples of features of the sound that can be controlled to provide anawareness of state include tempo, pitch, density, filter cut-off, rate,frequency, tonality, formants, randomness, articulation, loudness,phase, timbre morphing, voice multiplicity, and spatializationparameters, each potentially involving utilization of a distinctgeneration algorithm. Different combinations of attributes can be mappedto convey information. Also, different algorithms to manipulate theseperceived attributes can be used in different contexts.

In illustrative embodiments, a musical communication system isconfigured to map different synthesis techniques to parameters thatdefine models for specific use cases. A wide variety of differentsynthesis techniques can be used to convey information (e.g., currentstate with respect to goal state, amount of change and direction ofchange) through sound in illustrative embodiments disclosed hereininclude the following:

1. Density. Density can refer to the implied number of things happeningin a particular soundscape. A particular value could be mapped to howdense a perceived sound is. This falls into the category of a relativesound cue. Density can be a very effective musical tactic, as listenerswe get very excited by a sound that gets increasingly dense because itimplies growth and success.

2. High/low notes. This is a very specific system utilized in some sonicnavigation embodiments. For example, we can have two sounds: a high noteor complex sound in high range, and a low note or complex sound in anoticeably lower range. This is used to map a bidirectional sound, suchas “warm/cold” or “good/bad” or “left/right”. We use this in somenavigation and gesture models.

3. Rate/Tempo. Rate can be formally explained as a perception based onhearing the distance between a rapid succession of note onsets. Tempo isan often used subset of rate. We will often perceive rate as a kind ofspeed, such as fast or slow. Rate can also be used locally in thingslike clickers as a kind of continuous control. Vibrato can be a certainform of rate.

4. Frequency mapping. This specifically refers to taking data anddirectly mapping it to frequency in an oscillator.

5. Tonal/Noise. This is a subset of timbral morphing that involves goingbetween something perceived as pitched/tonal, to something that is morein the category of broadband noise. These two extremes provide veryclear morphing between states.

6. Formant filters/synthesis. Formants are a primary example of soundsthat are “absolute”: without any prior introduction, a listener couldeasily differentiate major vowel sounds such as “a”, “e”, “i”, “o”, and“u”. We emphasize different formants using filters and cross fadeparameters (filter cut-off, Q, band-width) to morph between vowels.

7. Filter Sweeps. Subtractive synthesis techniques that usually involvesshaping the timbre of a harmonically rich sound with a filter. Filtersprovide a very intuitive form of relative sound control, and workespecially well to communicate signals with regular dynamic movement.

8. Pitch Height. Pitch height refers to the relative distances betweennotes and refers to the range between the highest and lowest notesplayed during a given amount of time. This can be done sequentially formelodies, or in parallel for harmony.

9. Randomness. Randomness is a relative concept that refers to theamount of perceivable chaos in a system: Steady to unsteady, regular toirregular, predictable to unpredictable. This concept can be mapped tomany musical concepts like note duration and frequency/pitch selection.

10. Articulation. Articulation refers to the amplitude characteristicsof the beginning, middle, and end of a note envelope. This also extendsto note duration as well. Articulation is often nuanced.

11. Perceptual loudness. This includes amplitude modulation and directmapping to loudness. This one is also relative since sound systems havevolume control.

12. Phase Alignment. This refers to our ability to hear things out ofsync. For example, there may be two pulsating sound sources: one movingat a constant rate, and one that is controlled. The idea is it can betuned it to match the rate of the reference. This can work at sub audiorate signals with clicks/pulses, but also with tones.

13. Discrete Alerts. We define an alert as being an abrupt or new soundthat appears in the soundscape. The very presence of the sound is thecore means of conveying information. This is indeed an absolute sound,and a very clear one at that. Because of its binary nature, it is a verylimited mechanic.

14. Timbre Morphing. This is a broad category that includesinterpolation or crossfading between two or more distinct spectrums.Several techniques described above implement subsets of this approach.Timbral states are absolute sounds, and morphing between them is arelative sound.

15. Spatialization. Spatialization includes anything built to leveragethe ability of our ears to localize sound objects in a space. Primarilythis includes panning (left/right), object position using head-relatedtransfer functions (HRTFs), and/or reverberation. An HRTF, which caninclude an anatomical transfer function, is illustratively a responsethat characterizes how an ear receives a sound from a point in space.

16. Timbral Heterogeneity. This is a parameter that makes multipleperceived voices sound like a single source. An example algorithm usingthis parameter maps multiple voices to a parameter value which controlsthe timbre. At one extreme, all the voices share the same timbre withthe same spectrum, while at the other extreme they all have differentcontrasting timbral spectrums.

As mentioned previously, these and other particular features ofillustrative embodiments are presented by way of example only, andshould not be viewed as limiting in any way.

The above-described logic engine and audio engine of a musicalcommunication system as disclosed herein are illustratively implementedutilizing at least one corresponding processing device comprising aprocessor coupled to a memory. The processor executes software programcode stored in the memory in order to control the performance ofprocessing operations and other functionality. The processing devicealso comprises a network interface that supports communication over oneor more networks.

Still further illustrative embodiments will now be described withreference to FIGS. 30 through 32, which show aspects of goal-drivenauditory display techniques for use in breathing entrainmentsonification and other contexts involving musical communication and/orsonification as disclosed herein.

It is a common practice in exercise machines, wearables and otherfitness applications to employ goal-driven exercises to help a userquantify his or her progress over time to get a sense of how well he orshe is doing. Common exercise goals in conventional practice involve atime duration, exercise intensity, calories burned, distance traveled,heart rate, exercise cadence, or some combination of these.

The illustrative embodiments to be described include methods, apparatus,systems and computer program products for goal-driven auditory displaytechniques. The disclosed techniques are particularly well-suited foruse in enhancing user experience and performance in contexts such ascardio fitness, aerobic activity, and numerous others.

We have found that conventional approaches of the type mentionedpreviously are deficient in many important respects. For example,existing motivational systems for cardio fitness include DJ applicationsconsisting of precomposed content launched at different points in time.Such systems may function by matching a given running rate to songtempo. The systems either choose songs, or parts of songs in the form ofloops, which match the tempo of the music to the cadence of theexercise, i.e. steps per minute, or cycles per minute. These systems mayguide the user to perform the exercise at a certain rate, or match tothe rate of the exercise. As mentioned, these systems rely onprecomposed content that is launched at different points in time and donot generate or adapt content while it is playing, to the environment.

The music featured in these systems does not accurately andrealistically convey sufficient information about the user's progress orcurrent status with respect to a given exercise goal. For example, theinformation is not embedded into musical structures. If progress isconveyed acoustically, the cues are restricted to voice samples, i.e.where a recorded voice tells the user how he or she is doing, or amarker is triggered to signify to the user that he or she has passedsome check hold or reached some threshold.

Accordingly, while conventional systems exist to help users pace theirexercises using tempo, illustrative embodiments disclosed herein providesignificant improvements which overcome the above-noted drawbacks ofthese and other conventional systems.

For example, some embodiments disclosed herein are illustrativelyconfigured to use music to intuitively convey to a user, throughcontinuous sound cues embedded into parametric musical structures,controlled in real-time, how close he or she is to a fitness goal.

These and other illustrative embodiments disclosed herein are in somecases implemented in the form of an application that employs multipleauditory display and gamification techniques (e.g., levels) to enhanceperception, awareness of technique and motivation during cardio exerciseor other aerobic activity. The application can run on a mobiletelephone, computer, fitness equipment or other processing device.

Although the following illustrative embodiments are described primarilyin the contexts of cardio fitness and aerobic activity, it will bereadily apparent that the disclosed techniques can be adapted for use inother contexts involving a wide variety of other types of activities.

A goal-driven auditory display system in some embodiments disclosedherein may comprise one or more components, features or functionalitiesof an entrainment sonification system and/or a musical communicationsystem of the type described elsewhere herein.

In some embodiments, a goal-driven auditory display technique utilizes amixed approach with sonification, layering and “earcons” (e.g., soundcues) to help users meet exercise goals in cardio fitness, aerobicactivity and other contexts. For example, a goal-driven auditory displaysystem in such an embodiment is configured to adapt real-time audio toinput data to perform the following functions:

1. Convey to a user the current status of his or her progress withrespect to an exercise goal.

2. Motivate the user to perform a movement-based activity a certain wayto reach goal.

3. Guide user to perform activity at a certain rate (pace user). Ratecan influence the intensity of exercise.

4. Break long-term goal into shorter sub-goal sections which may beperceived as levels.

5. Parametrically reward a user for doing an activity correctly byincrementing a global parameter to provide a continuous control signalmapping to pitch, frequency, timbre, note probabilities, timing andamount of sound objects introduced into the system.

6. Create an illusion to a user that his or her activity (and energyexerted) is building the song and driving its progression forward.

FIG. 30 shows an example of ramp-up behavior in an illustrativeembodiment. If an activity detected satisfies necessary conditions, theglobal control parameter increments as shown by the upward diagonalarrows in the figure. An upward trajectory of a global control parameterdrives the musical progression forward, to signal to the user to keepdoing what he or she is doing. At the endpoint of each such upwardtrajectory, an earcon is generated as shown. Other than in the case ofthe final upward trajectory, a new level is then started from theendpoint, with a new upward trajectory, corresponding to a differentsub-goal of a main exercise goal. The different levels of increasingdifficulty in this example include levels A, B, C and D, each associatedwith a different instance of musical structure, progression and layeringas shown. Upon completion of the upward trajectory of the last level D,a final earcon is generated to indicate completion of the main exercisegoal.

It is apparent from the FIG. 30 embodiment that the musical structureprogression and layering change from level to level, as the userprogresses through the levels, which are of increasingly higherdifficulty. This is an example of what is more generally referred toherein as “goal-driven auditory display,” where auditory display in thisembodiment illustratively refers to varying the presentation of audio toa user in accordance with a progress curve and associated globalparameter values. The audio can be presented via a mobile telephone,tablet computer, fitness equipment, or other type of processing device,illustratively through associated speakers, earphones, wireless earbuds,etc.

FIG. 31 shows another illustrative embodiment, in this case utilizing aramp-up model with optional recovery sections. Such recovery sections insome embodiments illustratively include “cool-down” sections.

Recovery sections in illustrative embodiments are characterized bysparse orchestration (e.g., characterized by the beginning of a levelbefore global control parameter has been incremented) and a lack ofresponsiveness of the global control parameter to activity. Lack ofresponsive behavior and sparse orchestration signals to the user that heor she should take a break from intense exertion.

During recovery/cool-down sections, the incrementing build is unaffectedby movement-based activity. In addition, the recovery section will notincrease in pitch and brightness and will feature sparser density thanthe ramp-up sections. Finally, discrete auditory events (e.g., earcons)and/or variations in note patterns may be featured to signal thebeginning or the end of a recovery section.

The particular ramp arrangements used in the embodiments of FIGS. 30 and31 are presented by way of illustrative example only, and alternativeglobal parameter progress curve shapes can be used in other embodiments.

FIG. 32 illustrates global parameter behavior in an illustrativeembodiment. In this embodiment, an example global parameter has a valuethat ranges between 0.0 and 1.0, and that increments or decrements basedon the conditions of the activity detected. User levels up on the firstdown beat after reaching the maximum value (e.g., beat quantization isfeatured to preserve musicality).

The process flow in this example includes blocks 3200 through 3210 asshown. In block 3200, the system checks if the activity detectedsatisfies one or more conditions. For example, the system may determineif movement amplitude is above a specified threshold. If the amplitudeis greater than the threshold, as indicated in block 3202, the systemincrements the global parameter value in block 3203. If the amplitude isless than the threshold, as indicated in block 3204, or otherwise notgreater than the threshold (i.e., equal to the threshold) the systemdecrements the global parameter value in block 3205. In block 3210, ifthe global parameter reaches 1.0, the system waits until the musicaldownbeat to level up.

Additional examples of goal-driven auditory display techniques ofillustrative embodiments will now be described in further detail.

Some embodiments utilize entrainment (e.g., tempo pacing). A feature insuch embodiments illustratively comprises a tempo controlled rhythmiccomponent. For example, overall clock signal rate controls the timing ofrhythmic patterns featuring probabilistic structures which may beinfluenced by a substantially continuous ramp. Other types of rampshapes or arrangements can be used.

1. Exercises, such as interval training, often feature consecutivesegments that vary in intensity. Lower intensity is usually correlatedwith slower speeds and longer cycles or strides.

2. Intensity of a movement-based exercise is correlated with speed.

3. Users naturally entrain to repeating rhythmic structures, therefore,speeding up and slowing down the rate of clock guides users to adjustthe exercise intensity.

4. Entrainment via a tempo pacing feature helps users train better,perform better and have shorter recovery times.

Some embodiments utilize levels (e.g., for motivation). For example,gamification tactics can be employed with an auditory display techniqueto reward a user for performing an activity (correctly) over a durationof time. When activity is detected, a global parameter value from 0-1increments. Incrementing behavior speeds up if the activity detected ismore correct, i.e., closer to an optimal state.

A long term goal is split up into smaller goals which can perceived aslevels, to mark progress of the user and allow the user to have a moreclear sense of where he or she is with respect to goal. Shorter goalsallow for changes in sound to happen fast enough for the user toperceive the effect of his or her energy exertion in real-time.

The number of levels (e.g., sections) that a long term goal is split upinto can vary in different exercises.

Some embodiments utilize continuous controls, in which, for example,each level features continuous ramping up behavior (parametric reward),which is the sonification of current position relative to goal. This isa type of “progress” auxiliary sound cue of the type described elsewhereherein, and provides a sense of magnitude.

Continuous parameters answer questions the user may have such as “How amI doing?” and “How close am I from my target goal?”

Sonification techniques are selected in these embodiments to evokeimagery of a device “charging or powering up,” to give the user thefeeling that his or her activity is paying off. Sound parameter mappingsinclude rhythmic density, pitch and timbre. The progress information isconverted to a continuous audio-rate signal, where it is thenduplicated, scaled, and mapped to various components in the sound model.

Rhythmic density may be controlled via probabilistic sequences in orderto build up anticipation and motivation.

Pitch or frequency mappings, such as to filter cutoff of a resonant orpeaky filter, or to an oscillator, can be used to map out overalltrajectory.

Uniquely identifiable target spectra, such as vowel formants,characterize the beginning and end of ramp-ups which are interpolatedbetween.

Mappings to levels of auxiliary time-based effects such as delay linesor reverbs can provide a sensation of positive growth.

Additionally or alternatively, some embodiments utilize discretecontrols, as will now be described.

For example, a layering approach can be implemented in which, after auser completes a level, a new instrument, or sound component, isintroduced to increase the spectrum density and create a sense ofreward. This reward metric, provides a clear sound cue for the user todistinguish a higher level from a lower level.

As another example, a note pattern played by an existing instrument canbe varied from level to level, for example, the same instrument in levelone may play a different melodic sequence in level two. This is a lessobvious level distinguishing cue.

Another example of discrete controls involves the use of earcons. An“earcon” as the term is broadly used herein is intended to encompass anyof a wide variety of different types of brief, distinctive sounds thatare used to represent a specific event or to convey other information.For example, an earcon can comprise a discrete reward sound triggered atthe start of each level, as illustrated in FIG. 30, signaling to theuser that he or she has just completed a segment. The earcon can beimplemented as an audio file sample or a DSP sub-component. It may actas a placeholder for a sonic logo for branding purposes, e.g. once alevel is completed, a user hears a particular company's signature sound.Earcons in illustrative embodiments can therefore be customized fordifferent fitness companies or other providers.

Goal-driven exercise examples will now be described.

Different goal-driven exercises involve various inputs that define agoal, and thus goals can be defined by different actions where a systemmeasures the state of an activity and compares it to a set of rulesdefining a goal.

1. Time duration

-   -   Goal=perform an activity in a certain about of time.    -   Example 1: Global parameter is incremented by the simple passing        of time.    -   Example 2: Global parameter is incremented if any activity is        detected.

2. Distance

-   -   Goal=traveling across a real or virtual distance.    -   Global parameter increments if the user gets closer to the        distance goal.

3. Calorie

-   -   Goal=perform exercise to burn certain number of calories.    -   Global parameter increments if calories count rises.

4. Activity Intensity (e.g., one or more of resistance, incline, speedor other metric associated with exercise intensity)

-   -   Goal=Maintaining a certain intensity over a duration of time,        e.g. maintaining an amount of movement amplitude.    -   Global parameter increments if an intensity (movement amplitude)        detected matches or is above a certain intensity threshold.    -   Global parameter decrements if activity detected is below a        certain intensity threshold.    -   Target intensity may be varied throughout exercise.

5. Heart Rate

-   -   Goal=Maintaining a heart rate in a target zone over a duration        of time.    -   Global parameter increments if heart rate is in the target zone.    -   Global parameter decrements if heart rate is outside of target        zone.    -   Entrainment component is be manipulated to modify exercise        intensity to help user reach goal.    -   Additional adjustment cues may be employed to indicate to the        uses if he or she is above or below the target zone.    -   Target heart rate may vary throughout the exercise.

6. Activity Rate

-   -   Goal=Maintaining a target rate, amplitude or speed of movement        over a duration of time.    -   Goal=Entrain to the beat.    -   Global parameter increments if a rate of movement detected        matches the target rate.    -   Target rate is indicated by entrainment component rate/speed.    -   Target intensity may vary throughout the exercise.    -   Additional adjustment cues may be employed to indicate to the        user if he or she is above or below the target zone.

7. Running Technique Metrics (e.g., may include one or more of impact,asymmetry, bounce, ground contact time, stride length, gait, strikingdiscrepancy, GCT, flight time, contact time, step time and other runningor aerobic exercise technique metrics)

-   -   Goal=keep technique metric in optimal zone.    -   Global parameter increments if metric is in the target zone.    -   Global parameter decrements if metric is outside of target zone.

The foregoing are only examples, and numerous alternative arrangementsare possible. For example, a more complex exercise goal may featurecombinations of multiple goals which in sum to act as a normalized macrogoal, like an “exercise score,” consisting of multiple weighted metricscombined to influence the global parameter.

As indicated previously, the above-described entrainment sonificationsystem and/or musical communication system components, features andfunctionalities can be implemented in goal-driven auditory displaysystems in a wide variety of different contexts.

Some of the embodiments described herein implement closed-loop systemsthat feed physiological state data back to the user. Other embodimentsare implemented as open-loop systems that are configured to guidebreathing or other activities without feeding physiological state databack to the user.

In some breathing entrainment systems disclosed herein, breath may beguided using sound and music with features that synchronize with atarget breathing rate and indicate respiration phases. It may alsoinclude a breath signal customized by the user.

Open-loop entrainment may be achieved in some embodiments by mappingtarget breathing trajectory to sonic gestures. Such a system can utilizeone or more static files of audio/visual/haptic content. Closed-loopsystems present information about a user's physiological state, whichmay be a measurement of a stress-related physiological marker, inaddition implementing certain functionality associated with open-loopentrainment. The feedback may communicate to the user information abouthis or her performance during an exercise, and provide additional cuesto guide the user to a specified goal.

Some embodiments are configured to provide non-verbal sonic cues,possibly in combination with haptic cues or other types of cues. Suchcues can enhance breathing exercise instruction without interfering withvisual attention. Abstract sounds used to guide the breath cyclecomplement vocal instruction, through audio cues which provide moreresolution to vocal instruction. Practicing exercises using anentrainment sonification system as disclosed herein can enhancelong-term memory of pace, because temporal sequences are learned moreefficiently through sound than visuals. In this way, breathingexercises, guided by abstract sound and music, can encode breathingpatterns into musical memory.

In some embodiments, a breathing exercise makes use of a ratio whichcharacterizes the shape of the entrainment signal. The entrainmentsignal in such an embodiment may be generated by a measurement of theuser's natural breath cycle, or may be generated by a computer. Theparameters given below can be used to configure the entrainment content.An interface is used to allow the user to define these parameters atdifferent points in the exercise and automate the state to changegradually during a section. An example basic set of parametersillustratively includes the following:

1. Breathing Rate: The speed of the breathing entrainment signal. May becompared to tempo.

2. Ratio: The respiration signal shape defining the inner mechanics ofeach section of the breath cycle, e.g., the inhale, exhale and pausetimes. An inhale or exhale may have different subsections as well, forexample, an inhale may have two parts, where the user must first breatheinto the belly, then into the upper chest.

3. Duration: The length of the exercise or section of exercise.

4. Depth: How deep the breath is, from shallow, to deep. May be viewedas an amplitude of the breathing entrainment component.

In other embodiments, additional parameters are added to furthercustomize the breathing guidance provided by a breathing entrainmentsonification system. These additional parameters illustratively includethe following:

1. Timer: Musical mappings denoting the passing of time. They may bediscrete, such as transient sounds which mark different time durations.The timer may also automate the slow evolution of a parameter, forexample, slowly change the coefficients of a filter for an ambient soundwhich is slowly modulating from one state to another. The timer may beused to aid time management tasks, and is in particular useful for anapplication of this technology for increasing productivity in the workplace.

2. Voice guide/vocals: The abstract breathing guide sound to be coupledwith a recorded or artificially generated voice giving verbalinstructions of mental imagery or further defining the breathingtechnique. This illustratively includes vocals sparsity, a probabilisticmetric controlling how often voice is heard, and voice timing andplacement randomization, in which different sections of voice recordingsmay be tagged and assigned to different sections, where differentoptions may be selected based at least in part on a probability. Withsuch a system, many variations can be created on an exercise.

3. User Listening preferences: Other sound customizations which allowthe user to choose a style, generate a preset, or design a preset.

4. Masking: Type of user listening preference adding noise to block outunwanted background noise.

5. Mood or Intensity: This illustratively allows intensity or mood to bevaried dynamically through automation. Examples include pulsing beat,brightness, tonality, etc.

A backend portion of a given system as disclosed herein can beconfigured to generate audio files embodying a particular breathingexercise for a specific intention, through various combinations of thefeatures specified above, and possibly utilizing additional oralternative features. In such embodiments, an entrainment component maybe featured without an auxiliary component. Accordingly, it is to beappreciated that although some embodiments herein utilize both anentrainment component and one or more auxiliary components, otherembodiments can be configured to utilize only an entrainment component.As indicated above, an embodiment of the latter type can be configuredto map the entrainment component to abstract sound and/or music whichguides breathing for user.

In some embodiments, an adaptive system may be configured using aninteractive software application on a mobile telephone or computer. Suchan embodiment can include various entrainment features disclosed herein,including one or more of the following:

1. Biofeedback: An auxiliary component providing secondary cues back toa user to provide a sense of progress during the exercise.

2. Microphone Augmentation: A voice or breath recording may be used tocustomize the sounds featured in the breathing entrainment system. Forexample, a personalized breath sound may be used, in which the userrecords the sound of their breath sections, and the system captures thespectrum characterizing different points of the breath, and generates aunique filter which can capture some unique characteristics of thatperson's voice. This filter then would be used in the soundcharacterizing the breath guide. As another example, the user may singinto the microphone of a desktop, phone, tablet, or wearable device,which is then augmented with audio effects. The audio effects maymodulate the singing in different ways including: pitch shifting (toharmonize the singing), doubling (to make sound richer), distorting toadd overtones, reverb, delay, equalization, etc.

3. Interactive musical instruments: Casual interactivity with the soundfor further personalization.

Accordingly, some embodiments herein can be configured to provide basicbreathing entrainment solutions without auxiliary components, whileothers can provide more complex solutions incorporating one or moreauxiliary components, all in accordance with the techniques disclosedherein. For example, a basic solution can comprise static files createdusing a system backend in the manner described above, while a moreadvanced solution can feature an embedded augmented sound engine ininteractive software. The interactivity illustratively allows for thesound to be shaped while it is playing.

The above-noted basic solution can be implemented using packages ofaudio/video files featuring breathing entrainment content. A feedbacktrack may be generated or created to complement each different breathingexercise, and corresponding signals can be mixed in real-time tofacilitate a limited bio-feedback system.

Additional illustrative embodiments will now be described with referenceto FIG. 33. These embodiments provide example breathing exercises thatare implemented using breathing entrainment sonification systems of thetype disclosed herein. These examples include breathing exercisesdenoted as Exercise 1 through Exercise 9, each of which isillustratively guided by a breathing entrainment sonification system asdisclosed herein. FIG. 33 illustrates Exercise 1 as described below.

Exercise 1: Simple Breath

Intention=Calming

Duration=180 sec

Level of Expertise=Beginner

FIG. 33 shows the exercise timeline with parameters characterizingdifferent sections. The curves illustrate an entrainment signal overtime, guiding the breathing process of a user. Different sections arecharacterized by parameters at the beginning and end of each section,which are interpolated between throughout the exercise. The x-axis inthe diagrams of FIG. 33 illustratively depicts breathing rate inseconds, while the y-axis depicts volume/amplitude. In the first 90seconds the intention is to have the breather gradually shift from aninitial baseline of 12 breaths per minute (2 second inhale, 3 secondexhale) to a rate of 5 breaths per minute (6 second inhale, 6 secondexhale), with the volume also gradually increasing. For the next 60seconds, there is a sustained breathing rate of 5 breaths per minute (6second inhale, 6 second exhale), with a maximal volume of breath. Forthe last 30 seconds, the breather releases the technique and resumesnatural breath, which would very gradually taper down in its speed andvolume, but still remain slower and deeper than at baseline.

Exercise 2: Clearing Breath

Intention=Calming

Duration<1 minute

Level of Expertise=Beginner

The timeline, progression and sections in this example are as follows:

0-60 seconds: Take a big inhalation through the nose and exhale slowlythrough the mouth. Repeat two times.

Exercise 3: Intro to Paced Breathing (1:1 Ratio)

Intention=Calming & Focusing, also the foundational exercise upon whichthe subsequent examples are built.

Duration=3 minutes

Level of Expertise=Beginner

The timeline, progression and sections in this example are as follows:

0-60 seconds: Gradually shift from an initial baseline of 12 breaths perminute (2 second inhale, 3 second exhale, a typical involuntarybreathing pattern) to a rate of 5 breaths per minute (6 second inhale, 6second exhale), with the volume also gradually increasing.

60-150 seconds: Sustain breathing rate of 5 breaths per minute (6 secondinhale, 6 second exhale), with a maximal volume of breath.

150-180 seconds: Release the technique and resume natural breath which,over time, would gradually taper down in volume and increase in speed,yet still remain slower and deeper than at baseline. This period of timeat the end of each breathing exercise is intended for the practitionerto cultivate self-awareness by noticing the effects of the technique. Itis also intended to allow space for integration before practitionermoves on to the next task.

Exercise 4: Paced Breathing with Longer Exhalation (1:2 Ratio)

Intention=Calming

Duration=4 minutes

Level of Expertise=Intermediate

The timeline, progression and sections in this example are as follows:

0-60 seconds: Breather gradually shifts from an initial baseline of 12breaths per minute (2 second inhale, 3 second exhale—a typicalinvoluntary breathing pattern) to an even breathing 1:1 breathing ratio.Begin with 2 second inhale/2 second exhale, increasing to 3 secondinhale/3 second exhale, 4 second inhale/4 second exhale etc., up to 6second inhale/6 second exhale (5 breaths per minute). Volume willincrease as the breath lengthens.

60-90 seconds: Sustain a steady rate of 5 breaths per minute (6 secondinhale, 6 second exhale).

90-150 seconds: Gradually shift from a 1:1 ratio to a 1:2 ratio.

150-210 seconds: Sustain 1:2 ratio.

210-240 seconds: Release technique and return to involuntary breathing.Over time breath will gradually taper down in volume and increase inspeed.

Exercise 5: Intro to Resistance Breathing (for this exercise the focusis on learning the technique of resistance breathing. Other exercisescan combine resistance breathing with paced breathing.)

Intention=Calming

Duration=3 minutes

Level of Expertise=Beginner

The timeline, progression and sections in this example are as follows:

0-30 seconds: From involuntary, natural breathing gradually slow anddeepen the breath.

30-60 seconds: Verbal guidance to create a soft “ocean sound” on theexhalation by contracting the glottis (the swallowing muscle).

60-90 seconds: Verbal guidance to create the “ocean sound” on theinhalations as well.

90-150 seconds: Sustain the technique of creating a comfortable andsteady “ocean sound” on the inhalation and exhalation.

150-180 seconds: Release the contraction in the glottis, resume naturalbreath and observe the effects.

Exercise 6: Tactical Combat Breathing (1:1:1 Ratio)

Intention=Calming & Focusing

Duration=1 minute

Level of Expertise=Advanced Beginner

The timeline, progression and sections in this example are as follows:

0-4 seconds: Breathe in for 4 seconds.

4-8 seconds: Hold breath in for 4 seconds, staying relaxed in belly,shoulders, jaw, and throat.

8-12 seconds: Exhale for 4 seconds.

Repeat sequence 3-5 times.

Exercise 7: Alternate Nostril Breathing Prep

Intention=Focusing

Duration=4 minutes

Level of Expertise=Beginner

The timeline, progression and sections in this example are as follows:

0-30 seconds: From involuntary breathing, gradually transition to slowand deep nasal breathing.

60-210 seconds: Instruction given to seal one nostril and breathethrough only 1 nostril at a time, as follows:

-   -   Block off the right nostril with thumb or finger.    -   Inhale and exhale through the left nostril focusing on        maintaining a comfortable breathing rate & volume.    -   Repeat breathing in and out of left nostril 5 times total.    -   Release the right nostril and switch sides, using finger or        thumb to block of left nostril.    -   Inhale and exhale through the right nostril 5 times, maintaining        a comfortable breathing rate and volume.    -   Release the left nostril.    -   Take 5 breaths through both nostrils, maintaining a comfortable        breathing rate and volume.

210-240 seconds: Release the technique, resume natural breath andobserve the effects.

Exercise 8: Paced Breathing with Longer Inhalation (2:1 Ratio)

Intention=Energizing, but note that energizing breathing techniques mayspecify various contraindications and should be brief, typically no morethan 2-5 minutes. General contraindications include heart conditions,seizure disorders, pregnancy, recent surgery, hernia, bipolar disorder,etc.

Duration=4 minutes

Level of Expertise=Intermediate

The timeline, progression and sections in this example are as follows:

0-60 seconds: Breather gradually shifts from an initial baseline of 12breaths per minute (2 second inhale, 3 second exhale, a typicalinvoluntary breathing pattern) to an even breathing 1:1 breathing ratio.Begin with 2 second inhale/2 second exhale.

60-80 seconds: From baseline of 2 second inhale and 2 second exhale,gradually shift from a 1:1 ratio to a 2:1 ratio, inhaling for 4 andexhaling for 2.

80-120 seconds: Slowly increase the volume and duration of the breath,keeping the 2:1 ratio.

120-210 seconds: Sustain maximal 2:1 ratio.

210-240 seconds: Release technique and return to involuntary breathing.Over time breath will gradually taper down in volume and increase inspeed.

Exercise 9: Breathing with Segmented Inhalations

Intention=Energizing

Duration=1.5 minutes

Level of Expertise=Intermediate

The timeline, progression and sections in this example are as follows:

0-30 seconds: From involuntary breathing, gradually transition to slowand deep nasal breathing.

30-60 seconds: Verbal instruction to segment the inhalation into 3parts, each followed by a brief pause, then a continuous exhalation:

-   -   Inhale into lower lobes of lungs and pause.    -   Inhale into mid lobes of lungs and pause.    -   Inhale into upper third of lungs and pause.    -   Exhale naturally.    -   Repeat 5-10 rounds.

60-90 seconds: Release the technique, resume natural breath and observethe effects.

Illustrative breathing entrainment sonification systems as disclosedherein are configured to generate entrainment signals and possibly oneor more associated auxiliary signals for guiding a user through theparticular breathing patterns described above, in a manner that movesthe user toward the optimal pattern for each exercise over time. Suchsystems are therefore configured to translate the above-describedbreathing exercises into signals that are audibly presented to the userin order to achieve the desired entrainment functionality. Numerousother signal arrangements and exercise types can be used.

In some embodiments, additional information is provided to the user inconjunction with an entrainment component and possibly one or moreauxiliary components. For example, an audio configuration interface canbe configured to generate visuals mapping the entrainment component tosynchronized but evolving graphics which are either rendered into avideo, or ported to a game engine where they can be dynamicallycontrolled.

Different embodiments can incorporate different features andfunctionality based at least in part upon the particular user setting inwhich the system is deployed.

For example, some embodiments implement entrainment sonification systemson airplanes, illustratively as part of an inflight entertainmentsystem, or on a mobile telephone or tablet computer for use in thatsetting. Such embodiments can utilize static audio/video content or anembedded engine of the type disclosed herein, and may feature feedbackfrom heart sensor or EEG. Integrated content on embedded seats may beused.

In a corporate setting, users at desks can be equipped with an online,desktop or mobile telephone application, illustratively with featuresfor time management (e.g., timers) and masking. Meditation rooms orwellness pods may also be equipped with entrainment sonification systemsof the type disclosed herein.

It is also possible to implement entrainment sonification systems of thetype disclosed herein in yoga studios or other types of exercisesettings. For example, some embodiments can provide a real-time DJ appfor mixing breath music in user classes to enhance practice. Tracks usedin these embodiments may be generated beforehand or dynamicallycontrolled throughout class. Such a real-time DJ app can allow a classto breathe together in accordance with a desired pattern while providingvocal instruction simultaneously, and can be used in a wide variety ofdifferent exercise contexts, including, for example, flowing vinyasa(movement) and pranayama (still). Arrangements of this type can aid theinstructor by taking some strain off otherwise continuously having toconduct both breath and movement for the class. Breathing synced cues inmusic support better communication during practice and help foster asense of unity by keeping the class in rhythm.

In some embodiments, a mobile telephone application for at-home useprovides an interface for personalized meditation content creation,allowing an instructor to design individualized content to sharepractice with their followers, students or other clients. This is a toolthat will help them to generate perfect background tracks to fit guidedbreathing exercises, and to generate variations on recorded vocalinstruction. This meditation content creation tool allows instructors tocustomize practice for their clients. Content may be configured throughan application interface, allowing the instructor to define exerciseparameters, trajectory, and stylistic features making exercise unique.Content may be generated locally on the device, and/or on a cloudserver, then sent to individual users. Feedback may be an optionalfeature, if biometric data is available.

These and other embodiments can be deployed in spas, hotels, wellnesscenters, gyms, art galleries, airports, and a wide variety of othervenues. Some embodiments feature audio/visual content and optionalfeedback. Large quantities of tagged content can be stored in a databaseand looked up by metadata (e.g. intention, duration, etc.). Otherembodiments can use advanced adaptive integrated systems (e.g.,real-time training games). A wide variety of other implementations usingnumerous different platforms are possible, given the techniques of thepresent disclosure.

Entrainment sonification systems and other types of systems as disclosedherein, including by way of example musical communication systems and/orgoal-driven auditory display systems, are illustratively implemented atleast in part utilizing at least one corresponding processing devicecomprising a processor coupled to a memory. The processor is configuredto execute software program code stored in the memory in order tocontrol the performance of processing operations and otherfunctionality. The processing device also comprises a network interfacethat supports communication over one or more networks.

The processor may comprise, for example, a microprocessor, amicrocontroller, an application-specific integrated circuit (ASIC), afield-programmable gate array (FPGA), a central processing unit (CPU),an arithmetic logic unit (ALU), a digital signal processor (DSP), agraphics processing unit (GPU) or other similar processing devicecomponent, as well as other types and arrangements of processingcircuitry, in any combination.

The memory stores software program code for execution by the processorin implementing portions of the functionality of the processing device.A given such memory that stores such program code for execution by acorresponding processor is an example of what is more generally referredto herein as a processor-readable storage medium having program codeembodied therein, and may comprise, for example, electronic memory suchas SRAM, DRAM or other types of random access memory, read-only memory(ROM), flash memory, magnetic memory, optical memory, or other types ofstorage devices in any combination.

Articles of manufacture comprising such processor-readable storage mediaare considered embodiments of the invention. The term “article ofmanufacture” as used herein should be understood to exclude transitory,propagating signals.

Other types of computer program products comprising processor-readablestorage media can be implemented in other embodiments.

In addition, embodiments of the invention may be implemented in the formof integrated circuits comprising processing circuitry configured toimplement processing operations associated with the embodimentsdescribed herein.

Processing devices in a given embodiment can include, for example,laptop, tablet or desktop personal computers, mobile telephones, orother types of computers or communication devices, in any combination.

Communications between the various elements of an entrainmentsonification system comprising processing devices associated withrespective parties or other system entities may take place over one ormore networks. Such networks can illustratively include, for example, aglobal computer network such as the Internet, a wide area network (WAN),a local area network (LAN), a satellite network, a telephone or cablenetwork, a cellular network, a wireless network implemented using awireless protocol such as WiFi or WiMAX, or various portions orcombinations of these and other types of communication networks.

An entrainment sonification system as disclosed herein may beimplemented using one or more processing platforms, or portions thereof.

For example, one illustrative embodiment of a processing platform thatmay be used to implement at least a portion of an entrainmentsonification system comprises cloud infrastructure including virtualmachines implemented using a hypervisor that runs on physicalinfrastructure. Such virtual machines may comprise respective processingdevices that communicate with one another over one or more networks.

The cloud infrastructure in such an embodiment may further comprise oneor more sets of applications running on respective ones of the virtualmachines under the control of the hypervisor. It is also possible to usemultiple hypervisors each providing a set of virtual machines using atleast one underlying physical machine. Different sets of virtualmachines provided by one or more hypervisors may be utilized inconfiguring multiple instances of various components of the entrainmentsonification system.

Another illustrative embodiment of a processing platform that may beused to implement at least a portion of an entrainment sonificationsystem as disclosed herein comprises a plurality of processing deviceswhich communicate with one another over at least one network. Asindicated previously, the network may comprise any type of network,including by way of example a global computer network such as theInternet, a WAN, a LAN, a satellite network, a telephone or cablenetwork, a cellular network, a wireless network such as a WiFi or WiMAXnetwork, or various portions or combinations of these and other types ofnetworks.

Each processing device of the processing platform comprises a processorcoupled to a memory. As indicated above, the processor may comprise amicroprocessor, a microcontroller, an ASIC, an FPGA, a CPU, an ALU, aDSP, a GPU or other type of processing circuitry, as well as portions orcombinations of such circuitry elements. The memory may comprise RAM,ROM, flash memory or other types of memory, in any combination.

Again, the memory and other memories disclosed herein should be viewedas illustrative examples of what are more generally referred to as“processor-readable storage media” storing program code of one or moresoftware programs.

As mentioned previously, articles of manufacture comprising suchprocessor-readable storage media are considered embodiments of thepresent invention. A given such article of manufacture may comprise, forexample, a storage array, a storage disk, an integrated circuitcontaining RAM, ROM, flash memory or other electronic memory, or any ofa wide variety of other types of computer program products.

Also included in the processing device is network interface circuitry,which is used to interface the processing device with the network andother system components, and may comprise conventional transceivers.

Again, these particular processing platforms are presented by way ofexample only, and an entrainment sonification system, or other type ofsystem implementing musical communication and/or sonification techniquesas disclosed herein, may include additional or alternative processingplatforms, as well as numerous distinct processing platforms in anycombination, with each such platform comprising one or more computers,servers, storage devices or other processing devices. It is possible insome embodiments that system components can run at least in part incloud infrastructure or other types of virtualization infrastructure.

It should therefore be understood that in other embodiments differentarrangements of additional or alternative elements may be used. At leasta subset of these elements may be collectively implemented on a commonprocessing platform, or each such element may be implemented on aseparate processing platform.

Also, numerous other arrangements of computers, servers, storage devicesor other components are possible in an entrainment sonification system.Such components can communicate with other elements of the entrainmentsonification system over any type of network or other communicationmedia.

As indicated previously, components or functionality of the system asdisclosed herein can be implemented at least in part in the form of oneor more software programs stored in memory and executed by a processorof a processing device.

Accordingly, a given component of an entrainment sonification systemimplementing functionality as described herein is illustrativelyconfigured utilizing a corresponding processing device comprising aprocessor coupled to a memory. The processor executes program codestored in the memory in order to control the performance of processingoperations and other functionality. The processing device also comprisesa network interface that supports communication over one or morenetworks.

The particular configurations of entrainment sonification systems andother systems as described herein are exemplary only, and a given suchsystem in other embodiments may include other elements in addition to orin place of those specifically shown, including one or more elements ofa type commonly found in a conventional implementation of such a system.

For example, in some embodiments, an entrainment sonification system maybe configured to utilize the disclosed techniques to provide additionalor alternative functionality in other contexts. The disclosed techniquescan be similarly adapted for use in a wide variety of other types ofentrainment sonification systems.

It is also to be appreciated that the particular process steps used inthe embodiments described above are exemplary only, and otherembodiments can utilize different types and arrangements of processingoperations. For example, certain process steps described as beingperformed serially in the illustrative embodiments can in otherembodiments be performed at least in part in parallel with one another.

It should again be emphasized that the embodiments of the invention asdescribed herein are intended to be illustrative only. Other embodimentsof the invention can be implemented utilizing a wide variety ofdifferent types and arrangements of musical communication systems,entrainment sonification systems, or other information processingsystems, and associated networks and processing devices, than thoseutilized in the particular illustrative embodiments described herein,and in numerous alternative musical communication and/or sonificationrelated processing contexts. Also, the particular types andconfigurations of system entities, processing devices and processoperations can be varied in other embodiments. In addition, theparticular assumptions made herein in the context of describing aspectsof certain illustrative embodiments need not apply in other embodiments.These and numerous other alternative embodiments will be readilyapparent to those skilled in the art.

What is claimed is:
 1. An apparatus comprising: at least one processingdevice comprising a processor coupled to a memory; said at least oneprocessing device being configured: to generate a first sound cue of afirst type, the first sound cue comprising a primary entrainment cue foran entrainment sonification system; to generate one or more additionalsound cues of a second type, each of the one or more additional soundcues comprising an auxiliary entrainment cue for the entrainmentsonification system; to provide the first sound cue and the one or moreadditional sound cues to one or more audio devices of the entrainmentsonification system for generation of sound for audible presentation toa user; to receive from one or more sensors of the entrainmentsonification system one or more feedback signals; and to adjust one ormore characteristics of at least one of the first sound cue and the oneor more additional sound cues based at least in part on the one or morereceived feedback signals; wherein the entrainment sonification systemcomprises a breathing entrainment sonification system, and the firstsound cue comprises a primary breathing entrainment cue for thebreathing entrainment sonification system, the primary breathingentrainment cue comprising a particular entrainment signal configured todirect a breathing pattern of the user toward a desired breathingpattern.
 2. The apparatus of claim 1 wherein at least one of the one ormore additional sound cues comprises an auxiliary breathing entrainmentcue for the breathing entrainment sonification system, the auxiliarybreathing entrainment cue being configured to provide an indication tothe user of at least one of current status relative to a designated goaland guidance toward the designated goal.
 3. The apparatus of claim 1wherein the particular entrainment signal comprises a selected one of aplurality of distinct entrainment signals available within the breathingentrainment sonification system.
 4. The apparatus of claim 1 wherein oneor more characteristics of the particular entrainment signal areadjustable by a user.
 5. The apparatus of claim 1 wherein the particularentrainment signal comprises an entrainment signal having risingportions corresponding to respective inhale phases of the desiredbreathing pattern and falling portions corresponding to respectiveexhale phases of the desired breathing pattern.
 6. The apparatus ofclaim 1 wherein the particular entrainment signal comprises one of atriangular oscillator signal and a sinusoidal oscillator signal.
 7. Theapparatus of claim 1 wherein the particular entrainment signal comprisesa linear ramp signal with pause phases between adjacent instances ofrespective inhale and exhale phases.
 8. The apparatus of claim 7 whereinthe linear ramp signal comprises one or more variable air flow speedchanges.
 9. The apparatus of claim 1 wherein the particular entrainmentsignal comprises a complex ramp signal with at least one of one or morevariable trajectories and one or more external interrupts.
 10. Theapparatus of claim 1 wherein providing the first sound cue and the oneor more additional sound cues to one or more audio devices of theentrainment sonification system for generation of sound for audiblepresentation to a user comprises: combining respective signalsassociated with the first sound cue and the one or more additional soundcues in a signal combiner; and providing the resulting combined signalto an audio device.
 11. An apparatus comprising: at least one processingdevice comprising a processor coupled to a memory; said at least oneprocessing device being configured: to generate a first sound cue of afirst type, the first sound cue comprising a primary entrainment cue foran entrainment sonification system; to generate one or more additionalsound cues of a second type, each of the one or more additional soundcues comprising an auxiliary entrainment cue for the entrainmentsonification system; to provide the first sound cue and the one or moreadditional sound cues to one or more audio devices of the entrainmentsonification system for generation of sound for audible presentation toa user; to receive from one or more sensors of the entrainmentsonification system one or more feedback signals; and to adjust one ormore characteristics of at least one of the first sound cue and the oneor more additional sound cues based at least in part on the one or morereceived feedback signals; wherein the entrainment sonification systemis configured to implement an entrainment sonification valence/intensitymodel in which the primary entrainment cue is mapped to an intensityparameter, and the auxiliary entrainment cue is mapped to a valenceparameter.
 12. A method comprising: generating a first sound cue of afirst type, the first sound cue comprising a primary entrainment cue foran entrainment sonification system; generating one or more additionalsound cues of a second type, each of the one or more additional soundcues comprising an auxiliary entrainment cue for the entrainmentsonification system; providing the first sound cue and the one or moreadditional sound cues to one or more audio devices of the entrainmentsonification system for generation of sound for audible presentation toa user; receiving from one or more sensors of the entrainmentsonification system one or more feedback signals; and adjusting one ormore characteristics of at least one of the first sound cue and the oneor more additional sound cues based at least in part on the one or morereceived feedback signals; wherein the entrainment sonification systemcomprises a breathing entrainment sonification system, and the firstsound cue comprises a primary breathing entrainment cue for thebreathing entrainment sonification system, the primary breathingentrainment cue comprising a particular entrainment signal configured todirect a breathing pattern of the user toward a desired breathingpattern; and wherein the method is performed by at least one processingdevice comprising a processor coupled to a memory.
 13. The method ofclaim 12 wherein at least one of the one or more additional sound cuescomprises an auxiliary breathing entrainment cue for the breathingentrainment sonification system, the auxiliary breathing entrainment cuebeing configured to provide an indication to the user of at least one ofcurrent status relative to a designated goal and guidance toward thedesignated goal.
 14. A method comprising: generating a first sound cueof a first type, the first sound cue comprising a primary entrainmentcue for an entrainment sonification system; generating one or moreadditional sound cues of a second type, each of the one or moreadditional sound cues comprising an auxiliary entrainment cue for theentrainment sonification system; providing the first sound cue and theone or more additional sound cues to one or more audio devices of theentrainment sonification system for generation of sound for audiblepresentation to a user; receiving from one or more sensors of theentrainment sonification system one or more feedback signals; andadjusting one or more characteristics of at least one of the first soundcue and the one or more additional sound cues based at least in part onthe one or more received feedback signals; wherein the entrainmentsonification system is configured to implement an entrainmentsonification valence/intensity model in which the primary entrainmentcue is mapped to an intensity parameter, and the auxiliary entrainmentcue is mapped to a valence parameter; and wherein the method isperformed by at least one processing device comprising a processorcoupled to a memory.
 15. A computer program product comprising anon-transitory processor-readable storage medium having stored thereinprogram code of one or more software programs, wherein the program codewhen executed by at least one processing device causes said at least oneprocessing device: to generate a first sound cue of a first type, thefirst sound cue comprising a primary entrainment cue for an entrainmentsonification system; to generate one or more additional sound cues of asecond type, each of the one or more additional sound cues comprising anauxiliary entrainment cue for the entrainment sonification system; toprovide the first sound cue and the one or more additional sound cues toone or more audio devices of the entrainment sonification system forgeneration of sound for audible presentation to a user; to receive fromone or more sensors of the entrainment sonification system one or morefeedback signals; and to adjust one or more characteristics of at leastone of the first sound cue and the one or more additional sound cuesbased at least in part on the one or more received feedback signals;wherein the entrainment sonification system comprises a breathingentrainment sonification system, and the first sound cue comprises aprimary breathing entrainment cue for the breathing entrainmentsonification system, the primary breathing entrainment cue comprising aparticular entrainment signal configured to direct a breathing patternof the user toward a desired breathing pattern.
 16. The computer programproduct of claim 15 wherein at least one of the one or more additionalsound cues comprises an auxiliary breathing entrainment cue for thebreathing entrainment sonification system, the auxiliary breathingentrainment cue being configured to provide an indication to the user ofat least one of current status relative to a designated goal andguidance toward the designated goal.
 17. A computer program productcomprising a non-transitory processor-readable storage medium havingstored therein program code of one or more software programs, whereinthe program code when executed by at least one processing device causessaid at least one processing device: to generate a first sound cue of afirst type, the first sound cue comprising a primary entrainment cue foran entrainment sonification system; to generate one or more additionalsound cues of a second type, each of the one or more additional soundcues comprising an auxiliary entrainment cue for the entrainmentsonification system; to provide the first sound cue and the one or moreadditional sound cues to one or more audio devices of the entrainmentsonification system for generation of sound for audible presentation toa user; to receive from one or more sensors of the entrainmentsonification system one or more feedback signals; and to adjust one ormore characteristics of at least one of the first sound cue and the oneor more additional sound cues based at least in part on the one or morereceived feedback signals; wherein the entrainment sonification systemis configured to implement an entrainment sonification valence/intensitymodel in which the primary entrainment cue is mapped to an intensityparameter, and the auxiliary entrainment cue is mapped to a valenceparameter.
 18. The computer program product of claim 17 wherein thefirst sound cue comprises a primary entrainment cue for the entrainmentsonification system, the primary entrainment cue comprising a particularentrainment signal configured to direct the user toward a desiredbehavior.
 19. The computer program product of claim 18 wherein at leastone of the one or more additional sound cues comprises an auxiliaryentrainment cue for the entrainment sonification system, the auxiliaryentrainment cue being configured to provide an indication to the user ofat least one of current status relative to a designated goal andguidance toward the designated goal.
 20. The computer program product ofclaim 18 wherein the particular entrainment signal comprises a selectedone of a plurality of distinct entrainment signals available within theentrainment sonification system.
 21. The computer program product ofclaim 18 wherein one or more characteristics of the particularentrainment signal are adjustable by a user.